The scare

A search service Pip depends on starts timing out. Pip's code does the sensible-sounding thing: if it fails, try again. Immediately. Repeatedly.

Now imagine a hundred Pips. The service is struggling, and a hundred clients are each asking it again, and again, as fast as they can. Each retry is more work for a service that has none to spare. It falls over harder. Engineers call this a retry storm: the original problem was small and short, but the retries keep the outage alive.

This is not hypothetical. Amazon's engineers have written about exactly this in their Builders' Library. Their point about depth is the scary bit: if every layer of a system retries three times and the failing service is five layers down, it can end up seeing hundreds of times its normal load. Nobody planned that. Every layer just tried to be resilient.

Refresher: why retries can make things worse

Play with this. A hundred agents, one service that is down for the first three seconds. The dashed line is what it can handle; go far above it and it collapses instead of merely slowing.

Pick each strategy in turn. Retrying instantly, the service never recovers. Backoff on its own is better on paper, but everyone waits the same time, so they return in a synchronised wave. Adding jitter, a bit of randomness in the wait, breaks up the wave. That one small idea is the reason this simulation ends happily.

The pattern

A circuit breaker takes it further: stop calling altogether once a dependency is clearly down. The name comes from the fuse box. Too much current, the breaker trips, the wiring does not burn.

            failures reach threshold
   CLOSED ────────────────────────────► OPEN
   (calls flow)                         (calls refused instantly)
      ▲                                    │
      │ trial call succeeds                │ cooldown elapsed
      │                                    ▼
      └─────────────────────────────── HALF-OPEN
                trial call fails ──►   (one test call allowed)
                back to OPEN

Try it

Now you are the flaky API. Fail three calls to trip it, then wait out the cooldown.

Build it

import time

class CircuitOpen(Exception):
    pass

class Breaker:
    def __init__(self, threshold=3, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at = 0, None

    def call(self, fn, *args, **kwargs):
        if self.opened_at is not None:
            if time.time() - self.opened_at < self.cooldown:
                raise CircuitOpen("dependency is down; try the fallback")
            # cooldown over: half-open, let this one call through as a trial
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.opened_at is not None or self.failures >= self.threshold:
                self.opened_at = time.time()      # (re)open
            raise
        self.failures, self.opened_at = 0, None   # success closes it
        return result

search_breaker = Breaker(threshold=3, cooldown=30)

def flaky_search(q):
    raise TimeoutError("search API timed out")

for _ in range(5):
    try:
        search_breaker.call(flaky_search, "refund policy")
    except CircuitOpen as e:
        print("skipped:", e)          # calls 4 and 5 never touch the API
    except TimeoutError:
        print("failed")

Calls 4 and 5 never leave the building. The failing service gets breathing room, and Pip stops burning tokens on a lost cause.

What the agent does while it is open

An open breaker is information, so pass it on. Return "search is unavailable for about 30 seconds" and let Pip do something sensible: answer from what it already has, say so honestly, or take the next rung on the Fallback Ladder. What it must not do is loop.

When it goes wrong

• Threshold of 1. A service that hiccups once a day now blocks everything for 30 seconds each time.
• Nobody watches it. A breaker that opens silently looks like a mysterious slowdown. Log every state change to the trace.
• Counting the wrong failures. A 400 "bad request" is your bug, not the service's outage. Only trip on timeouts and 5xx.
• Retrying inside every layer. Retry in one place. See the depth problem above.

The agent-loop cousin

The same idea works inside the loop. If the agent has made the same call with the same arguments three times and got the same error each time, trip a breaker on that pattern. This is the practical end of Loop Engineering: a loop must always be able to stop.