Refresher: what "calibrated" means

Suppose a weather app says "70% chance of rain" on 100 different days. If it is calibrated, it rained on about 70 of them. If it rained on 45, the app is overconfident: it sounds surer than it has earned.

A typed-decision model does the same. When it says 0.9 on a ticket, you want it right about 90 times in 100 such tickets. That property is what lets you write "handle anything above 0.9 automatically". If the property is false, that rule quietly lets errors through.

Expected calibration error (ECE) is a single number for how far off the model is: group answers by stated confidence, compare each group's stated confidence to its real accuracy, average the gaps. Zero is perfect. A model can be very accurate and still badly calibrated, and the reverse.

What independent testers found

One study, by scienthoon, deliberately used 900 support tickets generated by a script on 19 September 2026, so the model could not have seen them, plus three public benchmarks. Here are the reported numbers.

The story is in the two tabs. On public benchmarks, which a model may have seen in training, the calibration error is small. On new tasks it is worse, and on the "priority" question, which depended on a company rule not present in the text, Jev averaged 0.74 confidence while being right 44.7% of the time. It was confidently guessing at something unknowable. (scienthoon)

The same study noted:

• Probabilities are quantised to 0.01 and often exactly 0 or 1, so a correct answer given zero probability cannot be recovered
• Choice and score answers were overconfident; yes/no was underconfident, so one fix does not suit all
• Do not threshold on the separate confidence field: it did worse than the highest probability
• Calibrate per question, not per model

Another reviewer found the same direction across studies (median ECE of 0.071 on a 13-subset public suite as shipped, with the direction varying by domain), and noted that one temperature fitted on 50 labels cut a comparison model's median ECE from 0.180 to 0.080 (xbill). Fifty labels is a small ask.

Calibrate it yourself

The fix is cheap: keep a labelled sample, and learn a correction. This code measures ECE on synthetic overconfident data, then fits one temperature on a small sample. It runs as written.

import math, random

random.seed(5)

def ece(probs, correct, bins=10):
    n, total = len(probs), 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        idx = [i for i, p in enumerate(probs) if lo <= p < hi or (b == bins - 1 and p >= hi)]
        if not idx:
            continue
        conf = sum(probs[i] for i in idx) / len(idx)
        acc = sum(correct[i] for i in idx) / len(idx)
        total += len(idx) / n * abs(acc - conf)
    return total

def scale(p, T):
    """Temperature: divide the log-odds by T. T above 1 makes it less confident."""
    p = min(max(p, 1e-4), 1 - 1e-4)
    return 1 / (1 + math.exp(-math.log(p / (1 - p)) / T))

# synthetic: stated confidence is high, but real accuracy is 15 points lower
def sample(n):
    probs = [min(0.999, 0.6 + 0.4 * random.random()) for _ in range(n)]
    correct = [1 if random.random() < max(0.0, p - 0.15) else 0 for p in probs]
    return probs, correct

fit_p, fit_c = sample(50)             # your small labelled sample
test_p, test_c = sample(2000)         # traffic you evaluate on

def nll(T):
    return -sum(math.log(scale(p, T) if c else 1 - scale(p, T)) for p, c in zip(fit_p, fit_c))

best_T = min([t / 10 for t in range(5, 60)], key=nll)
print("ECE as stated:      ", round(ece(test_p, test_c), 3))
print("fitted temperature: ", best_T)
print("ECE after fitting:  ", round(ece([scale(p, best_T) for p in test_p], test_c), 3))

Run it. The error falls once a temperature is fitted on just 50 labelled examples. On real data, do this per question type.

Wording is a release

A quieter finding from the same review: asking the identical question twice changed only 1.33% of answers, but swapping which rubric sat behind "yes" and "no" changed 32.5%.

Two practical rules follow:

1. Pin the model version you tested. The launch material mentions "Jev 1.13"; a silent version change is a silent behaviour change.
2. Every edit to your criteria is a release. Re-run your test set. This is exactly the Shadow Evaluation Harness rule.

A shadow test that would have caught the known failures

Run the typed-decision model beside your current process, on real traffic, deciding nothing. Then check:

Then a small canary behind an approval gate. One reviewer's whole pre-registered study ran 5,721 calls for $0.176 at list price (Beri). Testing is not the expensive part.

Sources

All pages read on 25 September 2026. Figures were pulled through a summarising fetch tool, so check any number on the linked page before you quote it.

• scienthoon, jev-ood-calibration. 900 out-of-distribution tickets plus three public benchmarks; per-type miscalibration; reproducible for about $0.06.
• xbill, Jev after eight days of independent tests. Median ECE, repeatability, the 32.5% option-swap result, language drops.
• Layer3 Labs, Jev benchmarks. What TypeSafe has not published; advice to validate on your own labels.
• Beri, phishing experiment. Shadow-evaluation advice and study cost.
• This site: Shadow Evaluation Harness, Trace Pipeline.