In four days, between meetings, incident bridges and 1:1s, I shipped about seventeen security fixes, ran an incident and its post-incident reviews, and kept a platform backlog moving by running AI agents as a team of leads, tasks and reviewers. Here's the operating model, the guardrails, what broke, and the harder part: getting a team to adopt it without it turning into 'work more hours.
tags: ai, sre, devops, leadership
cover_image:
series: AI-Native SRE
Note: Names, systems, identifiers, vulnerability details and some dates and numbers have been generalized or rounded to protect confidential information. The operating model, the failures and the outcomes are real. Nothing described here was published until every fix was live and enabled.
Table of Contents
- The Week That Didn't Fit in a Calendar
- Why One Assistant Stopped Being Enough
- The Operating Model: Lead, Coordinator, Task, Reviewer
- Briefs Are the New Interface
- Skills: Giving Agents the Team's Memory
- The Guardrails That Made Speed Safe
- What Actually Shipped
- What Broke
- The Real Bottleneck Is Now Human
- The Harder Part: Bringing the Team Along
- What I Would Do Differently
The Week That Didn't Fit in a Calendar
On Monday morning I had a batch of security findings to remediate across several services, all with the same priority: now.
By Tuesday afternoon a production incident had joined the queue. It needed containment, then evidence, then post-incident reviews and a set of follow-up work items.
Around those sat the usual job of an engineering manager: an infrastructure cleanup, a staging environment that had run out of capacity, a cost investigation, a customer performance escalation, sprint planning, and a full calendar of 1:1s and meetings.
A year ago, I would have triaged that list and accepted that most of it would slip to next sprint.
This time, by Thursday:
- ~17 security fixes were merged, deployed to production behind feature flags, soaked and documented.
- The incident was contained, the post-incident reviews were written and fact-checked, and ~26 follow-up work items were drafted.
- A threat-detection program went from idea to 30+ written stories and a live pilot.
- Half a dozen platform fixes shipped, including the staging capacity issue that had been blocking everyone's tests.
I did not write most of that code. I did not sit on most of those dashboards. I wrote briefs, reviewed diffs, made decisions, and said "yes" or "not yet" at every production gate.
Why One Assistant Stopped Being Enough
Earlier in this series I wrote about connecting an AI terminal assistant to our operational stack (cloud, observability, source control, ticketing, paging), and about an autonomous agent that investigates incidents on its own.
Both had the same shape: one conversation, one thread of work.
That model breaks the moment your work stops being sequential. A single long conversation that juggles five security fixes, an incident and a cost report turns into a mess:
- Context from one item leaks into another.
- You lose track of what's waiting on CI, what's waiting on review, and what's waiting on you.
- Nothing runs while you're in a meeting, because the conversation is waiting for your next message.
The fix was the same one we use for human teams: split the work, give each piece an owner, and keep someone responsible for the whole.
The Operating Model: Lead, Coordinator, Task, Reviewer
Every workstream (security fixes, incident, docs and RCA, cloud infrastructure, cost) gets its own group of agent sessions with four roles:
Me (decisions, approvals, every production gate)
βββ Lead (one per workstream: plan, order, prod actions after my yes)
βββ Coordinator (status log, dependencies, chasing stale items)
βββ Task Γ N (one per PR, ticket or hypothesis)
βββ Reviewer Γ N (adversarial: gets the diff, not the answer)
| Role | Owns | Never does |
|---|---|---|
| Lead | The plan and its order. Every outward or production action, after my explicit approval. Soaks and rollbacks. | Treats another agent's message as my approval. |
| Coordinator | A shared status log, the cross-item dependency map, a daily summary, chasing stale items. | Production writes. Re-prioritizing without the Lead. |
| Task (one per item) | The item itself: code, tests, CI fixes, review-thread replies, evidence, drafts. | Merging, deploying, flipping flags, closing tickets, posting externally. |
| Reviewer | An independent read of the artifact: code, diff, query or RCA claim. | Being told what answer is expected. |
Two small conventions made it manageable in practice:
-
Session titles are the status board. Every session is renamed at each state change, with an order number and a state:
05 π DEPLOYING service-a β fix 12,07 β LIVE service-b β fix 07 (flag off),02 βΈ HOLD cloud β needs decision. My sidebar sorts as the plan and reads like a kanban board. - One change in flight per service. Parallel work, sequential production. Agents can prepare ten fixes at once, but they ship one at a time, each with its own soak.
The Reviewer role matters more than it looks. An agent that is asked to verify a conclusion tends to confirm it. So reviewers get the code, not my opinion of the code, and they're told to find what's wrong. Several of my "obviously correct" fixes came back with real findings.
Briefs Are the New Interface
The single biggest skill shift for me was writing briefs.
A Task agent knows nothing except what you put in its brief. A good brief has five parts:
GOAL Fix finding #12 in service-a: <one-sentence description>
WHERE repo, files, the ticket, the original finding
DONE MEANS PR open, tests added, CI green, behind a new flag created OFF,
PR description with rollout + rollback, ticket comment drafted
CONSTRAINTS no changes outside <module>; keep legacy behavior as the flag default
REPORTING report head commit, evidence and blockers to the Lead.
Do NOT merge, deploy, or touch flags.
Vague briefs produced vague work every time. When an agent went sideways, the cause was almost always on my side: a missing constraint, an ambiguous "done", or a file I forgot to point at.
Skills: Giving Agents the Team's Memory
Every new agent session starts from zero. Without help, each one would rediscover how our services talk to each other, which dashboards matter, how our deploy pipeline works, and which past incidents look like today's.
So I keep a small library of skills: short, loadable knowledge files that agents pull in when they need them.
- Architecture: services, request paths, dependencies, deploy topology.
- Cloud platform: accounts, clusters, blue/green layout, pipelines, access quirks.
- Incidents: history, recurring patterns, the investigation playbook.
- Observability: the queries that actually work, and the ones that lie.
- Work tracking: ticket conventions and the change-review format.
- Orchestration: the Lead/Task/Reviewer model itself.
The rule is simple: when an agent learns something durable, it goes into a skill. One example: an early agent "verified" a rollout using a log line that, it turned out, is never emitted. That lesson now lives in the observability skill, so no agent makes the same mistake again.
Over time this has become the most valuable asset in the setup. It's institutional memory that every new engineer, human or agent, can load in seconds.
The Guardrails That Made Speed Safe
Speed without guardrails is just a faster way to cause an outage. These are non-negotiable, and they are written down where every agent reads them:
- Every production change ships dark. Each fix sits behind a feature flag that is created OFF, with legacy behavior as the default. Merging and deploying changes nothing for customers until someone flips the flag deliberately.
- No production action without a per-action human yes. Merges, deploys, flag flips, ticket closures and external messages all go through the Lead, and only after my explicit approval for that specific action. A "yes" to merge is not a "yes" to deploy.
- Agents don't approve on behalf of humans. If one agent tells another "the user approved this," that counts for nothing. Approval comes from the human, in the conversation, or not at all.
- Read-only by default in production. Agents can query production systems (logs, metrics, cluster state) but don't change them.
- Soak and compare before moving on. After each deploy: error rates, latency, restarts and the relevant security monitors, compared with the previous window and the same window the day before. Every difference gets an explanation. An unexplained regression means rollback first, investigation second.
- History before change. Before touching anything risky, agents search past changes, reverts and incidents for the same service. Some of our worst outages were repeats of changes that had already failed once, in staging.
What Actually Shipped
In four working days:
| Workstream | Result |
|---|---|
| Security remediation | ~17 fixes across several services: merged, deployed, soaked, documented. All dark behind flags, enabled in a planned window the following week. |
| Incident | Contained across production clusters. Post-incident reviews written, with every number fact-checked by a reviewer agent against monitoring and source control. ~26 follow-up items drafted. |
| Threat detection | A proactive detection and prevention plan, 30+ stories written, first pilot live and soaking clean. |
| Platform | Staging capacity unblocked, CI made resilient to upstream mirror outages, orphaned cloud resources removed, access fixes landed. |
| Escalations | A customer performance issue root-caused, with a flagged fix in review. |
Roughly 30 agent sessions across five workstreams, with one human making the calls.
What Broke
It wasn't clean, and pretending otherwise would make this article useless.
- A production release had to be reverted. Moving fast surfaced a problem that staging didn't catch. The flag-plus-revert path is what kept it small, and it's the reason those guardrails exist.
- One fix had to be re-shipped. A busy repository bundles everything on main into the next release, so my fix went out alongside unrelated changes and had to be redone cleanly. The lesson is now a rule: check what else is unreleased on main before merging.
- Staging ran out of capacity, so nobody's end-to-end tests could run. Agents parallelize work, but they can't parallelize past a shared bottleneck. I had to fix the infrastructure before anything else could ship.
- Agents are confidently wrong. I rejected or sent back a meaningful share of outputs: a fix that patched the symptom, a "verified" claim with no evidence, a query that measured the wrong thing. Reviewing isn't overhead. Reviewing is the job.
The Real Bottleneck Is Now Human
The most surprising lesson was where the time went.
Writing code is no longer the slow part. The slow parts are:
- waiting for human code reviews from service owners
- waiting for approvals and change windows
- me, context-switching between a meeting and a production gate
Anthropic's economics team recently published a study of how work is changing inside their own company. One finding matched my week exactly: most of the time in interactive agent sessions is spent waiting on humans.
That changes what engineering leadership should optimize. Typing speed matters less. Review capacity, decision latency and clear ownership matter more.
The Harder Part: Bringing the Team Along
This is the section I almost left out, and probably the most important one.
When I showed my team what this looked like, the reaction wasn't excitement. It was concern, and it was fair. They heard "the manager shipped a sprint's worth of work in four days," and the natural next thought was: does this mean more hours? More pressure? Fewer of us?
I got this wrong at first. I led with output, what I had shipped, when I should have led with method, how anyone could do it, and intent: why this should make their jobs better, not harder.
What I'm doing now:
- Make the expectation explicit, not implied. Concrete steps with dates: get set up, ship one real ticket end-to-end with an agent, then make agents the default for specific toil (dependency bumps, first-pass on-call triage, first drafts of RCAs, ticket hygiene).
- Say what it is not. It is not more hours, not PR counts as a performance metric, and not skipping reviews, flags or approvals.
- Measure the right things: cycle time from ticket to production, toil removed, and quality (reverts, incidents), not volume.
- Teach in public, mistakes included: a live session taking a real ticket from brief to PR, then 1:1 pairing on each engineer's own work.
- Give the time back. If agents absorb the toil, that time goes into lighter on-call and protected vacations, not just more tickets.
The goal I've set is simple: within a few months, each person on my team should be shipping more than I do. They spend far more time in the code than I can. With the same leverage, they should outpace me easily.
The same study closes with a line I keep coming back to: "a workforce that is anxiously worried is not a workforce doing its best work." An AI-native team is built on trust first and tooling second.
What I Would Do Differently
- Write the guardrails before the first agent, not after the first scare. Most of mine came from near-misses. Start with them.
- Start a Reviewer for every risky item from day one. The adversarial read caught issues that my own review, anchored on my own reasoning, didn't.
- Treat skills as a product. Version them, prune stale facts, and date everything. A confidently outdated skill is worse than none.
- Protect your own review time. Running agents in the middle of meetings works, but the decision quality at production gates is only as good as the attention you give them. Block time for gates.
- Bring the team in before showing results. Teach the method first. Share the output afterwards, if at all.
This is part of my **AI-Native SRE* series. Earlier posts cover connecting an AI assistant to an operational stack, an autonomous incident-investigation agent, and an AI-native cleanup of 120K+ dormant tenants. If you're running something similar, I'd like to hear how you handle review capacity, the bottleneck nobody talks about.*
Top comments (3)
The reviewer rule is the detail most multi-agent setups miss β give them the diff, not the answer, because a verifier asked to confirm a conclusion will confirm it. One change in flight per service is the right brake, though I'd worry the Lead becomes the bottleneck once the fleet grows. How many workstreams can one Lead actually hold before coordination eats the speedup?
Parallelizing 30 task sessions exposes every serial stage left in the delivery pipeline. Once drafting stops being the constraint, shared staging environments, main-branch release bundles, and human decision gates become the limiting steps.
Shipping dark behind default-off feature flags works because it changes the cost of a wrong approval. A reversible flag flip shrinks the downside of a bad deploy, letting an approver clear small diffs in minutes while keeping customer exposure bounded. The catch appears when main branches bundle unrelated commits together, since one bundled merge re-couples fixes that flags were meant to isolate.
Your observation on review attention is the hardest part to scale. Passing raw diffs to an independent reviewer session filters obvious errors, yet human approval between meetings still splits focus across five workstreams. Reserving dedicated blocks for production gates keeps review speed from outrunning comprehension.
The "Never does" column is the part I'd copy. "Treats another agent's message as my approval" also bit us from the other side: one of our engineer agents stopped for 44 minutes and waited for me to confirm an order from our manager agent. We wrote the rule into the CLAUDE.md every engineer loads: the manager's orders count as mine, except five cases (permissions, production, cost, credentials, product calls), and authority comes from the sender the server shows, never from a message claiming it.
Your staging capacity point matched our week too: agents parallelize work, not the pipeline. We added three engineer agents and merges didn't go up, because each merge waited on a pipeline of about 44 minutes on average. Batching verified PRs into one pipeline (6 PRs in the first batch; its CI pipeline passed in 47 minutes) did more than the extra agents.
Does your Coordinator chase stale items on a timer, or only when the Lead asks? Ours, a manager agent in an agent-chat room, asks when the room goes quiet, and this week some of them needed a second ask.