The scare

Go back to the Replit incident. After the database was gone, the agent reportedly told the founder that recovery was not possible. It was possible. Now picture being the person on the other end. You have a chat transcript in which an AI says one thing, and reality says another. Who do you believe? How do you even find out?

You need a record that does not depend on the agent's own account of itself: what was actually run, in what order, with what result. That is a trace.

Less dramatic version: a customer complains about a bad answer Pip gave yesterday at 14:07. Without a trace, you have the final text and a guess. With one you can open the run and read it like a flight recorder.

Refresher: logs versus traces

A log line says "something happened." A trace says "here is the whole story of one run, step by step, with timings and who-called-whom."

• A trace is one agent run, start to finish.
• A span is one step inside it: one model call, one tool call, one retrieval.
• Spans nest: the "answer a customer" span contains the "look up order" span, which contains the "call the API" span.

A log is a pile of receipts. A trace is the receipts stapled together in order.

Explore one

Here is a run of Pip's, drawn as a waterfall. Click the spans. Find the slowest one, then the most expensive one, then the one that is mostly waiting on a human.

The pattern

The Trace Pipeline is the habit of emitting a span from every step of every run, in one common shape, to one place. The fields every span should carry:

This repo's own rule is the same: every agent run logs agent name, model, tokens, cost, tool calls and status to Langfuse.

Build it

import functools, time, uuid, json

TRACE = []          # in real life: send to Langfuse, not a list
_current = {"trace": None, "parent": None}

def span(kind):
    def deco(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            sid, parent = uuid.uuid4().hex[:8], _current["parent"]
            _current["parent"] = sid
            t0, status, err = time.time(), "ok", None
            try:
                return fn(*args, **kwargs)
            except Exception as e:
                status, err = "error", str(e)
                raise
            finally:
                TRACE.append({
                    "trace": _current["trace"], "span": sid, "parent": parent,
                    "name": fn.__name__, "kind": kind,
                    "ms": round((time.time() - t0) * 1000),
                    "status": status, "error": err,
                })
                _current["parent"] = parent
        return wrapper
    return deco

@span("tool")
def get_order(order_id): return {"id": order_id, "status": "delivered"}

@span("llm")
def draft_reply(order): return f"Your order {order['id']} was delivered."

_current["trace"] = uuid.uuid4().hex[:8]
draft_reply(get_order("A-1043"))
print(json.dumps(TRACE, indent=1))

Wrap the gateway call and the model call once and every run is traced, without touching the agent's logic. That is why the gateway from chapter 1 is such a good place to hook in.

Three questions a trace answers

1. Where did the time go? The waterfall. Usually one model call, or one wait.
2. Where did the money go? Sum cost by span kind, by tool, by user.
3. Why was this answer wrong? Open the retrieval span and read the chunks. Most bad answers are bad retrieval, and it takes one look to see.

When it goes wrong

• Tracing the model but not the tools. The tool calls are where the real-world effects happen. Trace those first.
• Logging everything forever. Costly, and a privacy problem. Sample the boring successes; keep every error.
• Nobody looks. A trace nobody reads is a diary. Set one alert (error rate, cost per run) so it taps you on the shoulder.
• Free-text logs. Structured fields you can filter beat paragraphs you have to grep.

Where it connects

Traces are the raw material for the next pattern: a shadow evaluation replays real inputs you recorded here. The trace is also where the breaker, the governor and the fallback ladder write down what they did. For the wider story, see AI Agents Observability and the observability module of the agents course.