Building an agentic engineering team: running AI coding agents in a real engineering org

For the last couple of years most of my fractional CTO work has been some version of the same assignment: take a team that is experimenting with AI coding agents and turn that into a practice the whole org runs. I also do it on my own product. PeptIQ, which runs on iOS, Android and web, is built and operated almost entirely with an agent stack, and the live telemetry is on my now page. This post is what I have learned about making agents a working part of an engineering team rather than one engineer's fast laptop.

Start with what an agent is in your org

An AI coding agent is a contributor that works fast, never gets tired, does not remember yesterday, and will confidently do the wrong thing if the brief allows it. Treat it like a capable new hire with no context and no judgment about your business. That framing settles most of the arguments.

A new hire gets a written brief, a reviewer, a test suite that tells them when they broke something, limited production access, and a manager who looks at their output. Agents need exactly the same things. Teams that skip them get fast output and a slow, expensive cleanup.

The team shape

The agentic team I set up looks like this. Engineers own problems, not tickets. Each engineer runs one or more agents against the problems they own. The agents write most of the first-draft code. The engineer specifies, reviews, tests, and decides. Senior engineers also own the shared files that tell agents how the codebase works.

Nobody's title changes. What changes is the ratio of specifying and reviewing to typing. Engineers who are good at describing a problem precisely and spotting a wrong answer quickly become very productive. Engineers who were productive mainly because they typed fast have to learn a new skill, and the team needs to help them.

Specs before prompts

The single biggest change is that work starts with a written spec, not a chat message. The spec is short. One task, one owner. The exact system and files in scope. The acceptance criteria written as checks the agent can run. The actions it may not take. The proof it has to leave behind: tests run, files changed, a rollback note.

This feels slow for about a week. Then it becomes the thing that makes the output reviewable. A pull request that arrives with the spec, the tests it was written against, and a trail of what the agent did can be reviewed in minutes. One that arrives as a pile of diffs and a cheerful summary cannot.

Skill files: the codebase explains itself

Agents do not know your conventions. They do not know that your date handling lives in one module, that you never call the payments API from the client, or that the migration tool has a sharp edge. If that knowledge lives in senior engineers' heads, every agent run rediscovers it the hard way.

So we write it down in files the agent reads at the start of every task. Call them skill files, agent instructions, whatever you like. They cover the architecture in a page, the conventions that matter, the commands to build and test, the places where agents have caused problems before, and the rules for what needs a human.

Senior engineers own these files and they are reviewed like code. When an agent does something wrong that the file should have prevented, the fix is to the file, not just to the output. Over a few months the files become the most accurate documentation the team has ever had, because the agents punish every inaccuracy immediately.

Review gates that do not depend on attention

Human review does not scale to agent output volume. A team that doubles its pull request count and asks reviewers to look just as hard at each one gets either burned-out reviewers or rubber stamps. The answer is layers.

The first layer is automated and merciless: type checks, lint, the test suite, a build, and secret scanning on every change. Agents are not allowed to open a pull request that fails any of them. This catches most of the sloppy work before a person sees it.

The second layer is automated review by a model with a different brief than the one that wrote the code: look for logic errors, missing edge cases, security problems, and deviations from the skill files. It is not perfect. It is consistent, and it never skims.

The third layer is a person, and here the rule is about what, not whether. Every change gets a human reviewer. Changes to auth, payments, data models, infrastructure, and anything touching customer data get a senior reviewer and a second set of eyes. The engineer who owns the problem signs off on whether the change solves it. Nothing merges on agent say-so.

Testing: the agent writes the test first, and you check the test

Agents are good at writing tests and very good at writing tests that pass. Those are different things. The practice that works is to have the agent write failing tests against the acceptance criteria before touching implementation, then have the reviewer read the tests before reading the code. If the tests do not describe the behavior you asked for, stop there.

Keep the suite fast, because the agent will run it dozens of times per task. Keep flaky tests out, because an agent will happily "fix" a flaky test by weakening it. Add integration and end-to-end coverage for the paths that matter, because unit tests are what the agent is best at gaming.

Guardrails: what agents may not do

Write the forbidden list down and enforce it with permissions, not with hope.

Agents do not have production credentials. They do not run migrations against live data. They do not merge their own pull requests. They do not change CI configuration, dependency lockfiles for security-sensitive packages, or anything under the security and compliance directories without a named human approving. They operate in sandboxes with scoped tokens, and the scope is the minimum that the task needs.

Prompt injection is real. An agent that reads an issue, a web page, or a dependency's README can be told to do something by that content. Treat anything the agent reads from outside the repository as untrusted, limit what the agent can do with the result, and log it. The same SOC 2 instincts that apply to people, least privilege and auditable change flows, apply directly here.

Observability for the agents themselves

Once agents are producing a meaningful share of the code, you need to see what they are doing the way you see what production is doing.

Per task: which agent, which model, what it was asked, what it changed, what tests it ran, how many tokens and dollars it spent, and whether a human changed the output before merge. Per week: pull request count and size by source, review time, defect rate on agent-authored versus human-authored changes, and rollback count. Per month: spend by team and by task type.

This is the data that turns "AI is making us faster" from a feeling into a number, and it is also how you catch the failure modes early: an agent that is quietly generating enormous diffs, a model change that doubled the bug rate, a team whose spend tripled because a loop went wrong overnight.

Spend controls

Agent spend behaves like cloud spend. It is easy to start, invisible until the bill, and dominated by a few runaway cases. Give every agent a budget per task and per day. Route routine work to cheaper models and reserve the expensive ones for the tasks that need them. Report spend next to cycle time, so the conversation is about return rather than cost alone. And make someone own the number.

How I roll it out

The rollout I use with client teams has three phases, and I keep it to a quarter.

The baseline phase measures the team as it is: cycle time, review time, defect rate, the current state of tests and CI, and who is already using agents and how. We agree on the metrics we expect to move.

The install phase puts the pieces in: specs, skill files, the review layers, the permission model, the dashboards and the budgets. I pair with engineers on real tasks until the practice sticks, and I fix the skill files myself when the agents expose a gap.

The handoff phase reports the numbers against the baseline and transfers ownership. A senior engineer owns the skill files. An engineering manager owns the metrics. The team runs it without me. That last part is the point. The measure of success is proficiency across the whole team and shorter cycle time with the same or better quality, not a demo and not one hero engineer.

If your team is somewhere in the middle of this, the AI engineering leadership page describes the engagement, the fractional CTO page covers the broader role, my resume has the background, and the contact page is where to start the conversation.

Frequently asked questions

Should AI coding agents be allowed to merge their own pull requests?

No. Every change should pass automated checks and be approved by a named person who owns the problem. Agents can open pull requests, respond to review, and rerun tests, but the merge decision stays with a human.

How do you review code written by AI agents without slowing down?

Use layers. Automated checks and an automated review pass catch most problems before a person looks. Humans then review against the written spec and the tests, with senior reviewers required for auth, payments, data and infrastructure changes. Reviewing a spec and its tests is much faster than reverse engineering a diff.

What guardrails do AI coding agents need?

No production credentials, no live migrations, no self-merge, scoped tokens in sandboxes, a written list of files and actions that need human approval, and logging of everything the agent reads from outside the repository. Enforce them with permissions rather than instructions.

How do you measure whether coding agents are helping?

Track cycle time, review time, defect and rollback rates for agent-authored versus human-authored changes, and spend per task and per team, all against a baseline taken before the rollout. If quality holds and cycle time drops across the whole team, it is working.