The scare

Every pattern so far knows how to say no. The gateway refuses. The breaker opens. The governor stops the run. And each of those leaves the same awkward question: then what does the customer get?

If the answer is "an error page", you have built a harness that protects the system and abandons the person. Picture Pip on the day the model provider has a bad afternoon. Lumen & Co's customers are asking where their parcels are. The choices are a spinner forever, a cryptic error, or a message that says "I can't answer that right now, a person will reply by tomorrow." Only one of those keeps anyone's trust.

Refresher: graceful degradation

Lifts have this figured out. If the motor fails, the lift does not fall. It stops, brakes hold, lights stay on, an intercom works. There is a chain of "if that fails, then this", and the last link is deliberately simple.

Software calls it graceful degradation: get worse in steps, not all at once. Planning the steps ahead of time, calmly, beats improvising them during an outage.

The pattern

The Fallback Ladder is an ordered list of ways to still be useful, best first. Knock some rungs out and see where the request lands.

The rungs, in order:

1. Primary: the best model with all its tools
2. Retry with backoff: for temporary errors only
3. Secondary provider: a different provider's model. This is one reason every model call in this repo goes through the LLM gateway instead of hard-coding one vendor
4. Smaller model or cached answer: worse, but still an answer
5. Honest refusal and a human: say what happened and who will follow up. No dependencies, so it cannot fail

Retry only what is worth retrying

Build it

import random, time

class Retryable(Exception): pass

def with_backoff(fn, tries=3, base=0.5):
    for n in range(tries):
        try:
            return fn()
        except Retryable:
            if n == tries - 1:
                raise
            time.sleep(base * 2 ** n + random.random() * 0.1)   # jitter

def answer(question, providers, cache):
    for name, call in providers:                # ordered: primary, secondary, small
        try:
            return {"text": with_backoff(lambda: call(question)), "via": name}
        except Retryable:
            continue                             # next rung
    if question in cache:
        return {"text": cache[question], "via": "cache"}
    return {
        "text": "I cannot answer this right now. A person has been notified and will reply by tomorrow.",
        "via": "human_handoff",
    }

def down(_): raise Retryable("provider unavailable")
def ok(q): return f"Answer to: {q}"

print(answer("Where is order A-1043?", [("primary", down), ("secondary", ok)], {}))
print(answer("Where is order A-1043?", [("primary", down), ("secondary", down)], {}))

Every result carries via, the rung that answered. Log it to the trace. How often traffic lands below rung 1 is one of the best health signals you have. Give each rung its own circuit breaker, so a dead rung gets skipped instantly instead of retried.

When it goes wrong

• A ladder that has never been tested. Turn the primary off on purpose, in a quiet hour, and watch. Rungs nobody has walked tend to be broken.
• A fallback that costs three times as much. Pair the ladder with the governor.
• A fallback with different rules. The secondary model must go through the same gateway, permissions and context labelling. A backdoor rung is worse than none.
• A last rung that calls a model. The last rung must depend on nothing.

All ten, one request

  request
     │
  Context Boundary ──► label and filter inputs
     │
  agent loop ◄─────────── Cost and Rate Governor (budget, steps)
     │
  Tool Gateway ──► Permission Boundary ──► Approval Gate
     │                                        │
     ▼                                        ▼
  Sandboxed Runtime / real tools         human decides
     │
  Circuit Breaker on each dependency
     │
  Fallback Ladder if anything above fails
     │
  every step ──► Trace Pipeline ──► Shadow Evaluation (before the next release)

What to build first

You do not need all ten on day one. A sensible order:

1. Tool Gateway and Trace Pipeline: almost everything else plugs into these
2. Permission Boundary and Cost Governor: before the agent gets real power or runs unattended
3. Approval Gate: before it can do anything irreversible
4. Context Boundary: before it reads anything from outside
5. Circuit Breaker and Fallback Ladder: before real users depend on it
6. Sandbox: before it writes code
7. Shadow Evaluation: before your second release, which comes sooner than you think

The wider argument is in Harness Engineering, and the day-to-day running of all this is AI Ops.