The Clockwork Assumption

The alert fires, and for once there is a runbook.

That should be the good news. The primary database behind the checkout platform has started refusing writes in one region, orders are failing for every customer it serves, and the procedure for exactly this sits right there: numbered, precise, written by someone who lived through the last one. Somebody did the durable thing here. They took the time afterwards to write down what they had learned so the next person wouldn’t have to learn it again, the choice we keep saying we want people to make. It’s in her hands right now.

She starts at step 1.

Step 1 drains traffic out of the region. Step 2 promotes the standby database to primary. Both do exactly what the page says they will. Step 3 re-points the order service at the new primary. The command returns cleanly and the log line says what it always says. Step 4 restores traffic. Step 5 confirms the order service is writing to the new primary. It is.

Here’s what the procedure cannot know. When it was written, one service carried an order from payment through to shipping. Since then it has been split in two: authorization takes the card, and order dispatch releases it to the warehouse. A shared endpoint would have carried both halves. This platform, like most systems old enough to have a runbook, pins the address in each service’s own configuration. The runbook re-points what it was written to re-point, and dispatch is not in it. Nothing tells dispatch the database moved, so it keeps addressing the old primary, which has just been demoted and refuses every write. Step 5 checks the one service that existed when it was written. It went stale on the same day step 3 did.

She reaches the bottom of the page. Every step is done and every step returned cleanly, and the dashboard is still red. Payments are authorizing and nothing is shipping.

There is no step 3b. The procedure ran to completion and the outage is still happening. So now she’s scrambling, not to follow the runbook but to work out where it stopped being true, mid-incident, with the page still open in front of her and confidently describing the system as it was, not as it is.

The Right Plan, for a Different Time

The team that split the order service documented it and told the people they knew to tell. The main pages of the runbook were updated with the notice of the change. What escaped everyone was the embedded dependency buried in the break-glass procedure, a document owned by L1 production support, opened a couple of times a year and only under pressure. Somebody should have gone looking for it, and that is a real error. It is also the kind of error this arrangement will keep producing.

A bespoke process is brittle and breaks with even the most trivial change. Not a wrong plan. The right plan, for a different time. And the belief that makes it feel safe to write one down deserves a name: the Clockwork Assumption, that the part of the world a procedure describes will hold still for as long as the procedure is in force.

A clockwork is a precise machine, but that also makes it unforgiving. Every tooth is cut on the assumption that the next one sits exactly where it was when the movement was measured. There’s no slack in the mechanism, so nothing in it can absorb a change.

The obvious fix is to add the missing step. That closes this hole and leaves the procedure exactly as brittle as it was before: still a fixed sequence, still assuming the world will sit still, still unable to find out when it doesn’t. The next surprise is a renamed queue, or a new region, or a service that didn’t exist when the page was written, and the procedure will be just as blind to that one.

Hundreds of Small Mechanisms

Part 1 named the vacancy: the job that never gets staffed. Part 2 named the invoice: the Archaeology Tax, what that vacancy costs when a system finally has to change. Both are the same entropy wearing different clothes, the steady dispersal of what an organization once knew. Automation is a third face of it, and the one that collects fastest, because a script’s assumptions usually expire in the middle of an incident.

The typical organization is running hundreds of these entangled, brittle mechanisms. A scheduled batch job assumes an earlier batch job ran and finished clean. A pipeline depends on a naming convention that was never written down. A script calls a script that calls a script, and one of the three started life as somebody’s private crib notes and got promoted to production because it worked. The tendrils run between teams, between systems, and across years. Few organizations track this procedural glue at all.

The tangle explains a pattern most technology leaders have lived through: automation that looks excellent in year one and disappointing by year three. The savings were real. The upkeep went unplanned and unpriced, so maintaining the automation became its own backlog. The focus is almost always on what it saves against the current state, and not on the long tail.¹

A Pivot in Perspective

The first thing missing is being Goal-Oriented: given the outcome rather than the itinerary. “Get this region taking orders and shipping them again” is a goal. “Run these five commands in this order” is how we achieved it the last time. The difference only shows up when the two come apart, which is exactly what happened at step 3: the re-point returned cleanly and the service still failed. A script checks that the command succeeded. A goal checks that the thing you wanted is true. Those are usually the same check. When they aren’t, the script reports success and the outage continues.

The second missing property is being Adaptive: choosing what to do next based on what actually happened as a result of the action. A script cannot do that. When the expected behavior takes place, a script and an adaptive process reach the same outcome. When something unexpected happens, a script is a dead end. An adaptive process analyzes what went wrong, explores what it knows about the problem, forms a strategy, and tries again to achieve the stated goal. In short, it exercises judgment.

Give a process both properties and step 3 goes differently. It makes the same change the runbook makes, then checks the outcome: it compares the services writing to the new primary against the services that were writing to the database before the drain. Authorization is on both lists. Dispatch is only on the second, which means the change never reached it.

These two properties convert the static runbook into a system that is able to dynamically pivot as needed to achieve the desired outcome. The runbook stops when it reaches its last step. The process keeps going, because it was given the outcome and the outcome has not been reached. It re-points dispatch, checks again, and finds both services writing to the new primary. The outage ends at step 3, on a failure the runbook never described.

The static steps didn’t cover this failure scenario, because no one foresaw the services being split. Everything designed up front has that limit.

The Deterministic Layer

Plenty of work should stay deterministic. Reading a data record, copying a file, rotating a key on a schedule, running the same report and failing loudly when the columns move: work that is idempotent, reversible, and needs no judgment to do correctly. Call that the Deterministic Layer and leave it alone.

Right Twice a Day

The runbook didn’t fail her because it was badly written. It failed because it was cut to fit a world that had been measured once, and nothing in it could absorb the difference when the measurement changed.

The automation estate is not a library of solved problems. It is hundreds of mechanisms, each keeping perfect time, each built against a different version of the world, and all of them still running with complete confidence. Many of them are still right. The ones that aren’t will report success tonight in exactly the same voice.

Which turns the assumption inside out. The world isn’t the clockwork. The script is.

Our Knowledge Has a Slow Leak

That closes the case these three articles have been building. An empty chair at every fork, two weeks of digging before a one-line change, and a runbook that ran to completion while the outage continued. Three symptoms, one cause. The knowledge an organization has about itself leaks, slowly and continuously, and what remains disagrees with itself. The name for that is entropy, and it has been an opponent since the first program was written.

The leak is silent, and it compounds. The runbook is still mostly true, the wiki is only a little stale, the engineer who built the script still answers messages. Yet with each change, the organization finds itself less capable, often with no obvious culprit. Good people did good work the whole time and missed some small nuance. Then the primary fails over, and the nuance everyone missed is the reason nothing ships.

Leaky knowledge leads to brittle systems, and entropy is what causes it.

It would take superhuman effort to address every problem at its root, keep our documentation complete, and hold the nuance of every surrounding artifact in alignment. Entropy always wins in the end. While a system is running, the job is to push back faster than it leaks. We touched on Goal-Oriented and Adaptive behavior, which helps. Agents matter here for one reason: handed the goal instead of the steps, the thing doing the work is finally in a position to notice the world changed, though its judgment is still bounded by its understanding.

The more complete answer is a new way to manage what a team knows. Out of individual memory and decaying documents, into one place the whole team works from and keeps current as it works. What it takes to run a whole organization that way is the question the rest of this series takes up.

A principle for the pivot: When judgment will be required, encode the outcome you want over the steps that produce it.

Next in the series: what changes when the system gets a little better as it acts, instead of a little staler: The Quality Ratchet.

---

About this series: Next Quarter’s Normal is about how agentic systems are changing the way we build software. Today, though, we mostly apply agentic capability to assist our current practices. Useful, but not the point. The deeper opportunity sits in the problems we long ago filed under unsolvable, and in ways of working we haven’t imagined yet. We still build our tooling around the individual developer, when the heavy lift is getting an entire community to work in harmony. I don’t claim to have the answers; I’ve been building, and finding out. So each article closes by nominating one principle for the pivot: Waterfall to Agile took a re-imagination, and Agile to Agentic will too.

Footnotes

1. The research has the same blind spot. A 2022 review of the quantitative benefits of process automation screened 339 papers and analyzed the 98 carrying real numbers; between them they report savings, headcount, cycle time and payback, and nothing on what the automation cost to keep running. Many of those figures come from vendor case studies, which makes the omission more telling rather than less. The most-cited academic success case in that literature quotes a three-year return of 650 to 800 percent and never prices the third year of upkeep. The tail has been measured elsewhere. Across 265 OpenStack projects, infrastructure code changed in a given month about as often as the production code it deployed, and carried the highest churn per file of any category: a second codebase, not a finished asset. And across 45 months of change records on one large telephone switch, an equivalent change begun a year later took about 20 percent more effort, which compounds to roughly half again by year three. Ozturk et al., Robotic Process Automation: A Literature Review on Quantitative Benefits, IEOM 2022 https://ieomsociety.org/proceedings/2022istanbul/846.pdf; Lacity and Willcocks, Robotic Process Automation at Telefonica O2, MIS Quarterly Executive, 2016 https://eprints.lse.ac.uk/64516/1/OUWRPS_15_02_published.pdf; Jiang and Adams, Co-evolution of Infrastructure and Source Code, MSR 2015 https://mcislab.github.io/publications/2015/msrjojo.pdf; Graves and Mockus, Inferring Change Effort from Configuration Management Databases, IEEE METRICS 1998 https://mockus.us/papers/effort/. ↩