Harness Engineering

Here is a truth that surprises newcomers: in a real AI product, the model is maybe 10% of the work. The other 90% is the system built around it — the tools it can use, the memory it keeps, the limits on what it is allowed to do, the way you check if it is any good, and the way you see what it did. That surrounding system is the harness, and engineering it well is what separates a demo from a product.

A bare model takes text in and gives text out. That is all. To make it do something useful and trustworthy, you wrap it in machinery:

                    ┌───────────────── HARNESS ─────────────────┐
                    │  permissions   memory    observability     │
                    │       \          |          /              │
   user request ───────►   [  THE MODEL  ]   ───────► result     │
                    │       /          |          \              │
                    │    tools      runtime     evaluations      │
                    └────────────────────────────────────────────┘

Each piece of the harness has a job:

• Tools — what the model can do (search, calculate, call APIs) — from the RAG & Tools course.
• Memory — what it carries across steps — from the Context course.
• Permissions — what it is allowed to do.
• Runtime — how its actions actually run safely.
• Evaluations — how you know it is good.
• Observability — how you see what happened.

The next chapters take the three you have not met yet: permissions, evaluations, and observability.

Once a model can use tools, it can take actions — and actions have consequences. A support agent that can issue refunds could, if it misunderstands, issue a refund it should not. The protection is the same idea that protects human employees: least privilege — give it exactly the powers it needs for its job, and nothing more.

Practical permission patterns:

• Read vs. write — let it freely look up an order, but require a check before it cancels one.
• Limits — cap the size of any action (no refund over £100 without a human).
• Human-in-the-loop — for risky actions, the model proposes and a person approves.

You cannot improve what you cannot measure, and "it seemed fine when I tried it" is not measurement. Evaluations (evals) are a fixed set of test cases — questions with known good answers — that you run your system against to get a real score.

Why this matters so much: AI systems are easy to break by accident. You tweak a prompt to fix one thing and silently ruin five others. Without evals, you would never know. With them, every change gets scored against the same tests, so you can see did this actually help?

A simple eval set might be 50 real questions with ideal answers. You score the system on them (by exact match, by another model grading, or by a human spot-check) and watch that score as you make changes. The number going up is the only honest proof you improved.

When an AI system gives a strange answer or takes a wrong action, your first question is why? — and you can only answer that if you recorded what happened. Observability means logging the whole story of each request.

For every interaction, capture:

• What the user asked
• What context and documents were retrieved
• Which tools were called, with what inputs and results
• What the model produced — and how long it took and what it cost

This sets up the final stretch of the track. Once you can see what your system does, you can design the loop it runs (next course) and operate it safely in production (the course after). Observability is the foundation both stand on.