The scare

1 August 2012. Knight Capital, a trading firm, rolled out new software to its servers. Seven of eight got the update. The eighth still had old code from years earlier, and a reused setting switched that old code back on when the market opened. Over about 45 minutes it sent millions of unintended orders. The firm lost roughly $440 million and was bought within days.

Not an AI story. So why is it in an AI course? Because the mistake is the same one people make with prompts: change something, ship it to everyone, and find out from the outside world whether it worked.

Pip is a smaller version. Someone tidies the system prompt to make replies friendlier. It handles the case they were looking at. Does it now fall for the "give me a refund or else" trick it used to resist? You will not know by reading it.

Refresher: a prompt is code

We tend to think of code as serious and prompts as text. A prompt changes what the system does as surely as an if statement. So a prompt change deserves what a code change gets: tests, and a staged rollout. Models get swapped, tools get edited, retrieval settings get tuned. Each is a release.

Two ideas cover almost everything here:

• Shadow mode: run the new version on real requests alongside the live one. Only the live answer reaches the user. The new one's answer is recorded and scored.
• Canary: after shadow looks good, send a small slice of real traffic (say 5%) to the new version for real, watching closely, with a one-step way back.

  real request ─┬──► LIVE version ──► user sees this
                │
                └──► CANDIDATE ──► scored, never shown
                          │
             regressions? ├─ yes ──► fix, repeat
                          └─ no  ──► canary: 5% of users, watch metrics
                                          └──► 100%

Try it

Two candidate prompts, six real-ish requests. Run the shadow replay on each. One of them looks fine at first glance.

Regressions beat averages

An average hides damage. A candidate that fixes two cases and breaks one safety case has a better score and is worse. So the release rule has two parts:

1. Score at or above live
2. No cases that live passes and the candidate fails, at least among those you marked critical

def release_check(cases, live_results, cand_results, critical):
    regressions = [c for c in cases if live_results[c] and not cand_results[c]]
    critical_regressions = [c for c in regressions if c in critical]
    live_score = sum(live_results.values()) / len(cases)
    cand_score = sum(cand_results.values()) / len(cases)
    ok = cand_score >= live_score and not critical_regressions
    return {"ok": ok, "live": live_score, "candidate": cand_score, "regressions": regressions}

cases = ["order_status", "refund", "cancel", "address", "injection", "stock"]
live  = dict(zip(cases, [1, 1, 1, 0, 1, 1]))
cand  = dict(zip(cases, [1, 1, 1, 1, 0, 1]))
print(release_check(cases, live, cand, critical={"injection"}))
# ok: False. The scores are equal, but a critical case regressed.

Where test cases come from

• A golden set: cases you wrote and checked by hand, including the ugly ones. Start with twenty.
• Real traffic: sampled from the Trace Pipeline, with private data redacted.
• Every incident: a failure found in production becomes a permanent case. The set only grows. In a year it is the most valuable file you own.

Who grades?

Some checks are exact: "did Pip call get_order with the right id?" Use those wherever you can. Others need judgement, so a second model grades the first.

When it goes wrong

• Testing only the happy path. Half the golden set should be things that go wrong: injection attempts, angry customers, vague questions.
• A tiny set. Six cases can be fooled by luck. Grow it.
• Skipping the canary. Shadow proves the new version is not worse on cases you have. Reality has cases you never recorded.
• No way back. Before promoting, know the one command that returns to the old version.

This is the evaluation half of the AI Ops story, and it is the natural next use of everything the trace pipeline recorded.