Part one of a four-part series.
Every software organization sorts its systems into two classes. Production systems are instrumented, kept reliable, their changes controlled, their costs managed. Everything else is tooling: useful, complained about, and funded when the complaints get loud enough.
For decades, the feedback cycle of software development has lived in the second class. That cycle turns a change into a verified change and a shippable artifact — build systems, tests, dependency management, container builds — and it spans the entire pipeline, from the developer's machine to the pull request to the CI release build. The classification held not because it was right, but because its cost was invisible. AI agents make the cost visible, and much larger. Your toolchain is now a production system — whether it was built for that or not. The return on your AI investment depends on treating it like one.
Not whether customers touch it. Payroll faces no customer, and payroll is production. The test that matches how organizations actually behave: what do its failures cost, and who absorbs the cost?
Parts of your delivery stack passed that test years ago. When Git hosting or the CI service goes down, everyone stops, so it got high availability, backups, and an on-call rotation. But notice where the line ran: between down and degraded. Slow builds, flaky tests, wrong dependency resolutions — those got absorbed. The feedback cycle looked cheap because people made it look cheap.
The loop's problems never stopped anyone's work. A slow build? Context switch or coffee. A failure? Rerun it. A flaky test? "Ignore that one, it does that." Teams warped whole workflows around the pain: batching changes, avoiding the slow module, building before lunch.
Every one of these is a cost. None is a line item. The price was paid in fragments of salaried attention — four minutes here, a rerun there, across every developer, every day. Too small to be an incident, never aggregated by any meter. Finance cannot see inside a salary.
And there was nothing doing the seeing. The toolchain had almost no observability of its own: a local build left no record at all — when the terminal closed, it was gone — and CI kept whatever happened to land in a log. Instrumenting it properly was never justified, because it was not a production system. The circle closed: not production, so no observability; no observability, so no visible cost; no visible cost, so not production.
We spent a decade instrumenting these loops across thousands of organizations, and the pattern never varied: absorbed for years, invisible until someone measured it. As late as 2023, developers told GitHub they spent as much time waiting on builds and tests as writing code. The cost was always large enough to matter. It just never arrived as a bill.
Humans were the toolchain's shock absorber: salaried, patient, already paid for. Every failure looked free.
An agent runs the same loop — change, build, test — but its costs land in new places. The waste is metered now. Every false debugging path and every re-run to reproduce a failure is a paid round of reasoning on the token bill. Every wasted build and test run lands on the compute bill. And an agent has no tribal knowledge to repair bad signals. Your engineers know better than to trust a red build; agents do not. So a flaky test is no longer an eye-roll but a paid debugging cycle for a failure that never existed. And the meter does not distinguish waste from work: it all arrives as one bill.
What a waiting agent itself costs is genuinely unclear. The mechanisms are real: cached context expires and has to be rebuilt at resume, some harnesses poll while they wait, some hold billed sandboxes. But how much they amount to depends on the harness and the vendor, and both keep changing.
What accrues during the wait is perfectly clear: a change that waits long enough for the mainline to move underneath it has to be reconciled against a codebase that is no longer the one it was written against. A developer does that largely from memory of their own change. An agent pays to reconstruct the context from scratch: read both sides, re-derive the intent, re-run the verification. A slow loop keeps each change waiting longer. A busy fleet moves the mainline faster beneath it. The two multiply: twice the wait, against a mainline moving twice as fast, is four times the drift to reconcile. And the bill is the lesser consequence: the same drift limits how many agents can usefully run in parallel — it caps the fleet, and with it how much your agents can actually deliver. Part three will do the math.
The ratio inside the loop flips too. For a developer, thinking, reading, and writing dominated each iteration; agents collapse that side to minutes, while verification takes what it always took. The second-largest share of the delivery clock just became the largest.
And the volume multiplies: more iterations per task, fleets instead of one agent per developer, running overnight. If your agent rollout succeeds, I expect build and test volume to grow by an order of magnitude or more — at which point the compute line that was too small to matter in most organizations stops being too small. If it does not, you have a different problem.
The second-class treatment of the toolchain rested on cost that was absorbed and therefore illegible, or metered but too small to matter. That cover is gone. Absorption itself does not disappear; it moves — and part two will follow it.
Platform engineers object here: "Our CI has been production for a decade. We page on it." Correct — you page on its availability. The feedback cycle it runs is another matter: is there an SLO on build time, including local? A target for deployment frequency? A turnaround commitment on governance approvals? An error budget for flaky tests? An owner for the cost of a verified change? In most organizations, the honest answer is that availability got production treatment while the feedback cycle got tooling's budget with production's duty, and the gap was survivable because humans absorbed the difference. The absorbers are leaving the loop.
A system on the critical path of everything that ships, whose failures no one absorbs, is a production system by any operational definition. Nothing about your toolchain changed. Its consumers did.
The moment a system is classified as production, decades of settled operational consensus kick in — codified in production-readiness reviews and frameworks like AWS Well-Architected:
- It gets observed. An observability-grade record of every run, rich enough to reconstruct a root cause and to answer questions nobody anticipated when the build and tests ran.
- Its performance gets engineered. Latency targets and capacity planning for the loop, with an SLO on build time, because verification is now the largest share of the delivery clock.
- Its reliability gets engineered. Its consumers act on its every word, so a flaky signal is no longer an annoyance; it is corrupted ground truth, and someone gets paged when the signal degrades, not only when the service is down.
- Its changes get governed and its supply chain secured. Every artifact can prove how it was built, and policy holds at machine pace. Change volume that outruns human review still has to clear an audit, and a system that produces everything you ship is a target worth hardening.
- Its cost gets managed. An owner, attribution, and a watched ratio — useful output per dollar of compute and tokens — plus the ongoing engineering that improves it.
None of this is a new discipline. It is what you already do for every production system you run, applied to the one that just joined the class.
You do not have to take any of this on faith. Three numbers, all in your own systems, will settle it: local and CI build and test volume since your agent rollout, your failure rate including flaky tests, and the share of token spend going to retries and reruns. Then one drill: pick any change you shipped and time how long it takes to prove how it was built. If you do not have the numbers, or the proof takes weeks, that is a finding too. If the argument is right, the bill has already started arriving. It just has no owner yet.
Humans were the shock absorber, and humans still are. The absorption moved and concentrated, and it is now a ceiling on your agent program that no dashboard shows. Part two will cover where it moved.
Part three will work out the model: how the state of your toolchain affects lead time to merge, deployment frequency, and the compute and token bill — and the forces behind it.
The system that produces everything you ship is also your software supply chain. Securing it takes more than scanning what you deploy: evidence of how every artifact was built and what went into it, and strong identity for everything that takes part — because agents now author more change than human review can vouch for. Part four will take the production argument to supply chain security.
