The scare

This one is invented, but I would bet money you will recognise it. A team ships an agent on Friday. Over the weekend a customer's request hits an edge case: a tool keeps returning a slightly odd result, and the agent keeps trying again, slightly differently each time. Nothing crashes. No error alerts. It is technically "working". By Monday morning the API bill has a number on it that makes someone go quiet.

The original Knight Capital story from chapter 8 has the same shape at a horrifying scale: autonomous software, doing exactly what it was told, with no ceiling on how far it could go.

Refresher: why a loop costs more than you think

A chatbot costs one model call per message. An agent runs a loop, and each turn re-sends the whole history so far.

• Turn 1 sends about 2,000 tokens.
• Turn 2 sends 2,000 plus what happened in turn 1.
• Turn 10 sends maybe 6,000.

So ten turns cost far more than ten times turn one. Cost per step goes up as the loop runs, and total cost grows faster than the number of steps. And the model decides when to stop, as chapter 0 pointed out. If it never decides to, nothing else will.

Try it

A stuck agent, thirty steps if left alone. Turn the governor off and see what the loop costs. Then turn it on and move the budget and step cap.

The pattern

A governor is the part of an old steam engine that stops it running away with itself. Four limits, at four scales:

Set them before the agent runs, not after the first surprise. This repo's rule says the same: budget limits before any long-running task.

Build it

class BudgetExceeded(Exception):
    pass

class Governor:
    def __init__(self, max_usd=0.50, max_steps=12):
        self.max_usd, self.max_steps = max_usd, max_steps
        self.spent, self.steps = 0.0, 0

    def before_step(self, est_usd):
        if self.steps >= self.max_steps:
            raise BudgetExceeded(f"step cap {self.max_steps} reached")
        if self.spent + est_usd > self.max_usd:
            raise BudgetExceeded(f"next step would exceed ${self.max_usd:.2f}")

    def after_step(self, tokens_in, tokens_out, in_price=3e-6, out_price=15e-6):
        self.steps += 1
        self.spent += tokens_in * in_price + tokens_out * out_price

gov = Governor(max_usd=0.05, max_steps=8)
history_tokens = 2000
try:
    while True:
        gov.before_step(est_usd=history_tokens * 3e-6)
        # ... call the model here ...
        gov.after_step(history_tokens, 300)
        history_tokens += 400
except BudgetExceeded as e:
    print("stopped:", e, f"(spent ${gov.spent:.3f})")

The check happens before each step, using an estimate, so the run stops under budget rather than just over it. Prices vary by model and change often, so read them from config, not from memory. The model-cost comparison in Designing AI Applications shows how wide the gap between models is, which is why sending simple steps to a smaller model is a genuine cost lever.

What happens at the limit

Do not just crash. Choose on purpose:

1. Stop and report: "I used my budget. Here is what I found so far."
2. Ask to continue: hand it to the Approval Gate for a larger budget
3. Downgrade: finish the job on a cheaper model

When it goes wrong

• Only a daily cap. One stuck run can eat the whole day's budget before anyone notices. You need the per-run limit too.
• A limit so loose it never fires. If your average run costs 4 cents, a 50-cent cap protects against a stuck loop but a 20-dollar one protects against nothing.
• Counting only model spend. Tools cost money too: paid APIs, sandboxes, search.
• Silent stops. A run that ends at its limit should say so, and the event should show up in the trace.