The scenario

Kestrel Pay (our made-up payments app) gets 2 million support messages a month. Three things matter: send each to the right team, spot the angry or at-risk customers early, and never do anything irreversible without a person.

Today an LLM reads every message. It is accurate, slow, and expensive. Most messages are boring ("how do I change my email?"). A few are dangerous ("I don't recognise this transaction").

The design

  message ─► typed decision:  queue (choice) + at_risk (noul) + fraud_signal (noul)
                 │
                 ├─ fraud_signal high ──────────────► human, immediately (never automated)
                 │
                 ├─ queue confident AND low risk ───► route straight to the team
                 │
                 └─ anything else ──────────────────► LLM reads it, or a human does

Look at what is not automated: anything that could be fraud goes to a person no matter what the model's confidence says. The typed decision routes. It never decides money.

Try it: draw the line

This simulation gives 300 tickets a stated confidence each. Pick the line above which nobody double-checks. Then switch between "perfectly calibrated" and "overconfident" and watch the red squares.

That is the lesson in one screen: the line means what it says only if the confidence is honest. Independent tests found Jev's choice answers overconfident, and TypeSafe has not published a calibration measure. So your line has to come from your own labelled data, not from the number on the response.

Build it

Use the typed_decide.py stand-in from chapter 1.

from typed_decide import decide

QUESTIONS = {
    "queue": {"type": "choice", "criteria": {
        "billing":   ["charged", "refund", "invoice", "plan"],
        "account":   ["login", "password", "profile"],
        "technical": ["error", "crash", "bug"],
    }},
    "fraud": {"type": "noul", "criteria": ["don't recognise", "stolen", "unauthorised", "fraud"]},
}

AUTO_LINE = 0.80      # calibrate this on your own labelled data, never copy it
FRAUD_LINE = 0.20     # low on purpose: better a false alarm than a miss

def llm_triage(message):
    return {"route": "llm_review", "by": "llm"}          # stand-in for a real LLM call

def triage(message):
    a = decide(message, QUESTIONS)["answers"]
    if a["fraud"] >= FRAUD_LINE:
        return {"route": "human_now", "by": "rule", "why": f"fraud signal {a['fraud']}"}
    q = a["queue"]
    if q["confidence"] >= AUTO_LINE:
        return {"route": q["value"], "by": "typed", "confidence": q["confidence"]}
    return llm_triage(message)

for m in [
    "You charged me twice for the plan, please refund",
    "I don't recognise this transaction, my card was stolen",
    "Something is odd with my thing",
]:
    print(triage(m))

Read the output. The fraud message never reaches the auto route. The vague one falls through to the LLM. That is the whole pattern in 20 lines.

Where the tokens are saved

Only on the messages routed straight through. In the calculator, that was the "share handled alone". If half your traffic is routine, you avoid LLM input and output on that half. The saving is real, but it is exactly as big as the share you can safely automate, and that share depends on your data.

The rules that make it safe

1. Route, do not decide anything involving money, identity, or safety.
2. Set the escalation line from data. A few hundred labelled examples, split by confidence band.
3. Sample the auto-routed traffic weekly. A human reads 50. If errors creep up, tighten the line.
4. Log every decision with its confidence to the trace.
5. Have a fallback rung if the API is down: send everything to the LLM or to a queue, do not stop the support desk. See the fallback ladder.

What could go wrong

• A confident wrong route. A billing ticket that is really fraud lands in the wrong queue and waits. Keep the fraud line separate and low.
• Drift. Kestrel Pay launches a new product; new kinds of message arrive; accuracy slides quietly. Sampling catches it.
• Non-English customers. Independent tests saw a drop in Russian and a smaller one in Spanish. Test each language separately, or route non-English to the LLM.
• A prompt tweak that changes everything. One review saw a third of answers change after swapping the rubric wording. Treat criteria edits as releases (chapter 6).

Sources

All pages read on 25 September 2026. Figures were pulled through a summarising fetch tool, so check any number on the linked page before you quote it.

• xbill, independent tests. Few-class strength, spam parity, language drop, wording sensitivity.
• scienthoon, jev-ood-calibration. Overconfidence on choice answers; 900-ticket out-of-distribution test.
• Layer3 Labs, Jev benchmarks. What TypeSafe has and has not published.
• Agentic AI Harness Patterns (this site): Approval Gate, Fallback Ladder.