Course overview
A chatbot answers questions. An agent takes action.
When you ask a chatbot "What is the weather today?", it tells you what it knows from training. An agent, given the right tools, can actually call a weather API, read the result, and then answer you. If you ask it to book a meeting for a day with no rain this week, an agent can check the forecast for each day, look at your calendar, find an opening, and create the event — with no step-by-step instruction from you.
This course takes you from zero to a working multi-agent system, with full observability so you can trust what your agents are doing.
No prior agent experience needed. Familiarity with Python and basic LLM APIs (Claude, OpenAI) will help.
What is an AI Agent?
- Understand the key difference between a chatbot and an agent
- Know when agents are the right tool and when they are not
- Recognise the components that make up an agent: LLM, tools, memory, loop
The simplest definition
An AI agent is a system where a language model controls a loop — it takes in information, decides what to do next, acts (usually by calling a tool), observes the result, and repeats. It keeps going until the task is done or it runs out of options.
CHATBOT
┌────────────────────────────────────────────────┐
│ User: "What is the weather?" │
│ │ │
│ ▼ │
│ LLM: "I don't have real-time weather data." │
│ (done — one round trip) │
└────────────────────────────────────────────────┘
AGENT
┌────────────────────────────────────────────────┐
│ User: "What is the weather in Amsterdam?" │
│ │ │
│ ▼ │
│ Think: I should call the weather tool │
│ │ │
│ ▼ │
│ Act: weather_api("Amsterdam") │
│ │ │
│ ▼ │
│ Observe: {"temp": 14, "cond": "cloudy"} │
│ │ │
│ ▼ │
│ Think: I have the data. I can answer now. │
│ │ │
│ ▼ │
│ Answer: "It's 14 deg C and cloudy today." │
└────────────────────────────────────────────────┘
The four components of an agent
Agent = LLM + Tools + Memory + Loop
LLM — the reasoning engine. Decides what to do at each step.
Tools — functions the LLM can request (search, write, query, send). The LLM does not run them — you do. It just asks.
Memory — what the agent knows: the current conversation, past runs, or a knowledge base.
Loop — the cycle of Think → Act → Observe that repeats until the task is done.
Chatbot vs Agent: key differences
| Chatbot | Agent | |
|---|---|---|
| Input | A message | A goal |
| Output | A text response | A result (file, action, decision) |
| Steps | One | Many |
| Tools | None | Many |
| Autonomy | You drive | It drives (with guardrails) |
| Reliability | High, predictable | Needs observability |
When to use an agent
Use an agent when:
- The task has multiple steps that cannot be hardcoded
- The task needs real tools (search, write, query, send)
- The task can fail partway and needs to retry or adjust
- The steps are not known in advance
Do not use an agent when:
- A single LLM call is enough
- The steps are fixed — just chain prompts in code instead
- Low latency is critical (agents do multiple round trips)
- Errors are unrecoverable (agents can make wrong decisions)
- An agent is an LLM + tools + memory + loop
- A chatbot does one response; an agent loops until the task is done
- The LLM never runs tools directly — it requests them, and your code runs them
- Use agents for multi-step, dynamic tasks; use direct LLM calls for simple ones
- What are the four components of an agent?
- Name one scenario where you should NOT use an agent.
- Who actually runs the tool code — the LLM or your Python code?
The Agent Loop — ReAct
- Understand the ReAct pattern (Reason + Act)
- Follow a complete agent trace step by step
- Know what a stopping condition is and why it matters
ReAct: Reason + Act
The most common agent pattern is ReAct, introduced in a 2022 research paper. The model alternates between reasoning (what should I do?) and acting (call a tool or give a final answer).
User: "What is the weather in Amsterdam today?"
|
v
+---------------------------------------------------+
| Step 1 - THINK |
| "I need current weather data. I'll call the |
| weather tool with city=Amsterdam." |
+---------------------------------------------------+
|
v
+---------------------------------------------------+
| Step 2 - ACT |
| tools/call: weather(city="Amsterdam") |
| [your code runs the API call] |
+---------------------------------------------------+
|
v
+---------------------------------------------------+
| Step 3 - OBSERVE |
| Result: {"temp_c": 14, "condition": "cloudy"} |
+---------------------------------------------------+
|
v
+---------------------------------------------------+
| Step 4 - THINK |
| "I have the data. I can now answer the user." |
+---------------------------------------------------+
|
v
Answer: "It's 14 deg C and cloudy in Amsterdam today."
STOP: no more tool calls needed
Stopping conditions
An agent stops when:
- The model produces a final text answer instead of a tool call
- The model explicitly says the task is done
- You hit a maximum step limit (important safety measure)
- A budget limit is reached (token or cost cap)
Always set a step limit. Without one, a misbehaving agent can loop forever.
Inline reasoning ("extended thinking")
Some models (Claude 3.7+) support extended thinking — the model produces visible reasoning tokens before deciding what to do. These are not shown to the user but are visible in the API response and in observability tools. Extended thinking improves complex reasoning but costs more tokens.
- ReAct = Reason + Act — the standard agent loop pattern
- The loop repeats until the model gives a final answer or you hit a limit
- Always set a maximum step limit to prevent runaway agents
- Extended thinking lets you see the model's reasoning process in the API
Tools and Function Calling
- Define tools using the Anthropic API format
- Build the minimal agent loop from scratch in Python
- Apply the four principles of good tool design
How tool calling works
Every tool has three parts:
- A name the model uses to request it
- A description that tells the model when and why to use it
- A JSON schema describing what inputs it takes
1. You define tools + send user message to the API
|
v
2. Model responds with a tool call request:
{ "name": "get_weather", "input": { "city": "Amsterdam" } }
|
v
3. You run the actual function:
result = get_weather("Amsterdam") -> {"temp": 14, ...}
|
v
4. You send the result back to the model
|
v
5. Model produces next thought or final answer
|
v
Repeat from step 2 until final answer
The minimal agent loop
import anthropic
import json
client = anthropic.Anthropic()
# The actual function — runs on YOUR machine
def get_weather(city: str) -> dict:
mock = {
"Amsterdam": {"temp_c": 14, "condition": "Partly cloudy"},
"London": {"temp_c": 11, "condition": "Rainy"},
}
return mock.get(city, {"error": "City not found"})
# Tool definition — what the model reads
tools = [{
"name": "get_weather",
"description": "Get current weather for a city. Returns temp in Celsius and weather condition.",
"input_schema": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name, e.g. 'Amsterdam'"}
},
"required": ["city"]
}
}]
def run_agent(user_message: str) -> str:
messages = [{"role": "user", "content": user_message}]
while True:
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
tools=tools,
messages=messages,
)
# Model wants to call a tool
if response.stop_reason == "tool_use":
tool_block = next(b for b in response.content if b.type == "tool_use")
name, args = tool_block.name, tool_block.input
print(f" → Tool: {name}({args})")
result = get_weather(**args) if name == "get_weather" else {"error": "unknown tool"}
print(f" ← Result: {result}")
# Add exchange to conversation and loop
messages.append({"role": "assistant", "content": response.content})
messages.append({
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": tool_block.id, "content": json.dumps(result)}]
})
else:
# Model gave a final answer — we are done
return next(b.text for b in response.content if hasattr(b, "text"))
print(run_agent("What is the weather in Amsterdam right now?"))
Good tool design — four principles
1. Be specific in descriptions
| Bad | Good |
|---|---|
| "Gets files" | "List all files in /workspace and return names and sizes" |
| "Database thing" | "Run a SELECT query. Returns up to 50 rows. Only SELECT allowed." |
| "Search the web" | "Search DuckDuckGo and return the top 5 result titles and URLs" |
2. Return structured data — dicts and lists, not free-form strings. The model handles JSON-like data more reliably.
3. Handle errors gracefully — return {"error": "message"} instead of raising exceptions. The model can read the error and try a different approach.
4. Keep tools focused — one tool, one job. A tool that does five things is hard to use correctly.
- Every tool needs a name, description, and JSON schema
- The minimal agent loop: call API → handle tool request → run function → send result → repeat
- Good descriptions are the most important factor in reliable agent behaviour
- Return structured data and errors as dicts, not exceptions or plain strings
- Write a description for a tool that searches a product database by price range.
- What should a tool return when it fails — an exception or an error dict? Why?
- What happens after the model returns a
stop_reasonof"end_turn"instead of"tool_use"?
Single-Agent Patterns
- Build three single-agent patterns using LangGraph
- Understand when each pattern is appropriate
- Run a working research agent that searches and writes a file
Using LangGraph's built-in agent
create_react_agent gives you a full ReAct loop without writing it from scratch:
from langchain_anthropic import ChatAnthropic
from langgraph.prebuilt import create_react_agent
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.tools import tool
search = DuckDuckGoSearchRun()
@tool
def save_file(filename: str, content: str) -> str:
"""Save content to a file in the current directory."""
with open(filename, "w") as f:
f.write(content)
return f"Saved {len(content)} chars to {filename}"
model = ChatAnthropic(model="claude-3-5-sonnet-20241022")
agent = create_react_agent(model, tools=[search, save_file])
result = agent.invoke({
"messages": [("human",
"Research what MCP (Model Context Protocol) is, write a 300-word summary, "
"and save it to mcp-summary.txt")]
})
print(result["messages"][-1].content)
Install: pip install langgraph langchain-anthropic langchain-community duckduckgo-search
Pattern 1: The Research Agent
The research agent searches for information, reads it, and synthesises it. This is the most common single-agent use case.
Goal: "Research LangGraph and write a summary"
|
v
Think: I'll search for LangGraph
Act: search("LangGraph framework")
Observe: [3 search results with titles and snippets]
|
v
Think: The first result looks most relevant. I'll read it.
Act: fetch_page("https://...")
Observe: [article text, 2000 chars]
|
v
Think: I have enough. I'll write the summary and save it.
Act: save_file("summary.txt", "LangGraph is...")
Observe: "Saved 450 chars to summary.txt"
|
v
Answer: "I've saved a summary about LangGraph to summary.txt"
Pattern 2: The Router Agent
A router agent receives a message and decides which specialised handler to use. This is useful when you have multiple types of requests.
from langchain_core.tools import tool
@tool
def handle_writing(topic: str, length: str) -> str:
"""Handle a content writing request — blog post, email, or summary."""
return f"[Writing handler]: {length} piece about {topic}"
@tool
def handle_code(language: str, task: str) -> str:
"""Handle a code writing or explanation request."""
return f"[Code handler]: {language} — {task}"
@tool
def handle_research(query: str) -> str:
"""Handle a research or fact-finding request."""
return f"[Research handler]: {query}"
router = create_react_agent(model, tools=[handle_writing, handle_code, handle_research])
Each handler could be a full sub-agent in a more complex system.
Pattern 3: The Code Execution Agent
import subprocess
@tool
def run_python(code: str) -> str:
"""
Execute a Python code snippet and return its stdout output.
Safe, non-destructive operations only. No file writes or network calls.
Times out after 10 seconds.
"""
result = subprocess.run(
["python3", "-c", code],
capture_output=True, text=True, timeout=10
)
return result.stdout if result.returncode == 0 else f"Error:\n{result.stderr}"
create_react_agentbuilds the ReAct loop automatically — use it unless you need custom control- Research → Router → Code Execution are the three most common single-agent patterns
- Each pattern fits a different type of task — choose based on what the agent needs to decide at runtime
Memory — What Agents Remember
- Understand the four types of agent memory
- Know which type to use for which scenario
- Implement persistent key-value memory in Python
+--------------------------------------------------------------+
| Type | Persists? | Retrieval | Best for |
+--------------------------------------------------------------+
| In-context | No | Automatic | Current session |
| (conversation) | | | |
+--------------------------------------------------------------+
| Key-value | Yes | Exact key | User prefs, |
| (dict/JSON/Redis)| | | entity facts |
+--------------------------------------------------------------+
| Vector | Yes | Semantic | Large knowledge |
| (embeddings) | | similarity| bases, past runs |
+--------------------------------------------------------------+
| Episodic | Yes | By task | Learning from |
| (run log) | | or date | past outcomes |
+--------------------------------------------------------------+
Type 1 — In-context memory
This is simply the conversation history passed to the model. LangGraph manages this automatically in the messages state key. It disappears when the session ends.
Type 2 — Key-value memory (persistent)
import json
from pathlib import Path
MEMORY = Path("agent_memory.json")
def save_fact(key: str, value: str):
"""Save a fact to persistent memory."""
facts = json.loads(MEMORY.read_text()) if MEMORY.exists() else {}
facts[key] = value
MEMORY.write_text(json.dumps(facts, indent=2))
def recall_fact(key: str) -> str | None:
"""Recall a saved fact."""
if not MEMORY.exists():
return None
return json.loads(MEMORY.read_text()).get(key)
# Usage examples:
save_fact("user_timezone", "Europe/Amsterdam")
save_fact("user_language", "English")
recall_fact("user_timezone") # → "Europe/Amsterdam"
Type 3 — Vector (semantic) memory
Useful when you have too many facts to fit in context and need to retrieve the most relevant ones by meaning rather than exact key.
# Simplified example — in production use Pinecone, Qdrant, or pgvector
import chromadb
chroma = chromadb.Client()
collection = chroma.create_collection("agent_memory")
def store_memory(text: str, doc_id: str):
collection.add(documents=[text], ids=[doc_id])
def recall_similar(query: str, n: int = 3) -> list[str]:
results = collection.query(query_texts=[query], n_results=n)
return results["documents"][0]
Type 4 — Episodic memory
A log of past agent runs — what task was given, what steps were taken, what the outcome was. Useful for:
- Detecting when an agent keeps failing the same way
- Providing context to future runs ("last time you tried X, it failed because Y")
- Analytics on agent behaviour over time
import json, datetime
def log_episode(task: str, steps: list[str], outcome: str, cost_usd: float):
episode = {
"timestamp": datetime.datetime.utcnow().isoformat(),
"task": task,
"steps": steps,
"outcome": outcome,
"cost_usd": cost_usd,
}
with open("episodes.jsonl", "a") as f:
f.write(json.dumps(episode) + "\n")
- In-context memory: automatic, disappears at session end
- Key-value memory: persistent, fast, exact lookup — for user preferences and facts
- Vector memory: persistent, semantic retrieval — for large knowledge bases
- Episodic memory: a run log — for learning from past outcomes
Multi-Agent Systems
- Understand why multiple agents outperform a single agent on complex tasks
- Know the three core multi-agent patterns
- Build a 3-agent pipeline using LangGraph StateGraph
Why multiple agents?
A single agent handling everything has real limits:
- Context window — very long tasks overflow the context
- Specialisation — a generalist agent writes worse than one focused only on writing
- Reliability — independent agents checking each other catch more errors
- Parallelism — multiple agents can work on different parts simultaneously
The three patterns
PATTERN 1: ORCHESTRATOR + SUBAGENTS
-------------------------------------
User -> Orchestrator -+-> Researcher -> result
+-> Writer -> result
+-> Editor -> result
|
v
Final output
Best for: tasks where subtasks are known, can be delegated clearly
PATTERN 2: PIPELINE (fixed sequence)
-------------------------------------
User -> Researcher -> Summariser -> Writer -> Editor -> Output
Best for: tasks with a clear, fixed sequence of steps
PATTERN 3: SWARM (peer-to-peer)
-------------------------------------
User -> Agent A <-> Agent B <-> Agent C -> Output
|
Agent D
Best for: emergent, exploratory tasks with unknown path
Building a 3-agent pipeline with LangGraph
from typing import TypedDict, Annotated
import operator
from langgraph.graph import StateGraph, START, END
from langchain_anthropic import ChatAnthropic
from langchain_core.messages import HumanMessage, BaseMessage
class PipelineState(TypedDict):
topic: str
research: str
draft: str
messages: Annotated[list[BaseMessage], operator.add]
model = ChatAnthropic(model="claude-3-5-sonnet-20241022")
def researcher(state: PipelineState) -> PipelineState:
print("[Researcher] Working...")
response = model.invoke([HumanMessage(content=
f"Research '{state['topic']}'. Write structured notes covering key concepts, "
f"recent developments, and important examples. Be specific and factual."
)])
return {"research": response.content}
def writer(state: PipelineState) -> PipelineState:
print("[Writer] Drafting...")
response = model.invoke([HumanMessage(content=
f"Using these research notes, write a 400-600 word blog post about '{state['topic']}'.\n"
f"Include a clear title, 3-4 paragraphs, and a practical conclusion.\n\n"
f"Research:\n{state['research']}"
)])
return {"draft": response.content}
def editor(state: PipelineState) -> PipelineState:
print("[Editor] Reviewing...")
response = model.invoke([HumanMessage(content=
f"Edit this draft. Check for clarity, accuracy, and tone. "
f"Return the improved version at the same length.\n\nDraft:\n{state['draft']}"
)])
return {"draft": response.content}
# Build the graph
builder = StateGraph(PipelineState)
builder.add_node("researcher", researcher)
builder.add_node("writer", writer)
builder.add_node("editor", editor)
builder.add_edge(START, "researcher")
builder.add_edge("researcher", "writer")
builder.add_edge("writer", "editor")
builder.add_edge("editor", END)
pipeline = builder.compile()
result = pipeline.invoke({
"topic": "Why MCP changes how AI agents connect to tools",
"research": "", "draft": "", "messages": []
})
print(result["draft"])
START
|
v
+--------------+
| researcher | <- node (your Python function)
+------+-------+
| edge (data flows through)
v
+--------------+
| writer |
+------+-------+
|
v
+--------------+
| editor |
+------+-------+
|
v
END
Adding conditional routing
def needs_revision(state: PipelineState) -> str:
"""Return 'researcher' to loop back, or 'end' to finish."""
if "INSUFFICIENT_DATA" in state["draft"]:
return "researcher"
return END
builder.add_conditional_edges("editor", needs_revision, {
"researcher": "researcher",
END: END,
})
- Multiple agents beat single agents on complex tasks via specialisation and parallelism
- Orchestrator: one coordinator delegates to specialists
- Pipeline: fixed sequence, each agent hands off to the next
- Swarm: peer-to-peer, no central coordinator
- LangGraph makes the flow explicit — nodes are agents, edges are connections
- Give one reason why a multi-agent system might outperform a single agent.
- Which pattern would you use for a task with a fixed 4-step workflow?
- What is the difference between
add_edgeandadd_conditional_edgesin LangGraph?
Agent Frameworks
- Understand the strengths and trade-offs of six agent frameworks
- Run a code example for each major framework
- Know which framework to reach for based on your use case
Framework overview
Need quick experiment? -> Agno (cleanest API, fast setup)
Need role-based team? -> CrewAI (researcher + writer + reviewer roles)
Need code generation + exec? -> AutoGen (conversation + code runner built in)
Need reliable production flow? -> LangGraph (explicit graph, full control)
Learning how handoffs work? -> OpenAI Swarm (educational, not production)
Using open-source models? -> Smolagents (HuggingFace, code-first)
LangGraph
What it is: Graph-based framework where you define nodes (agents/functions) and edges (connections). Explicit, stateful, full control.
Best for: Production systems. Complex flows. Any use case needing human approval or exact control.
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
@tool
def search(query: str) -> str:
"""Search the web and return a summary of results."""
return f"[Search results for: {query}]"
agent = create_react_agent(
ChatAnthropic(model="claude-3-5-sonnet-20241022"),
tools=[search]
)
result = agent.invoke({"messages": [("human", "What is LangGraph?")]})
print(result["messages"][-1].content)
CrewAI
What it is: High-level framework where agents have roles, goals, and backstories — like job descriptions for a team.
Best for: Business process automation with clear roles (researcher, writer, reviewer, manager).
from crewai import Agent, Task, Crew, Process
researcher = Agent(
role="Research Analyst",
goal="Find accurate, up-to-date information on any topic",
backstory="Expert at synthesising information from multiple sources.",
llm="claude-3-5-sonnet-20241022",
verbose=True,
)
writer = Agent(
role="Technical Writer",
goal="Write clear, engaging content about technical topics",
backstory="Turns complex ideas into readable prose.",
llm="claude-3-5-sonnet-20241022",
)
research_task = Task(
description="Research AI agents. Find 5 key capabilities with examples.",
expected_output="Structured list of 5 capabilities with real examples.",
agent=researcher,
)
write_task = Task(
description="Write a 400-word blog post using the research.",
expected_output="Complete blog post with title and 3 paragraphs.",
agent=writer,
)
crew = Crew(
agents=[researcher, writer],
tasks=[research_task, write_task],
process=Process.sequential,
)
result = crew.kickoff()
print(result.raw)
Install: pip install crewai
AutoGen (Microsoft)
What it is: Conversation-based multi-agent framework. Agents talk to each other. A UserProxyAgent can run code locally.
Best for: Code generation + execution. Debate/verification patterns. Multi-agent conversations.
import autogen
config_list = [{"model": "claude-3-5-sonnet-20241022", "api_key": "your-key", "api_type": "anthropic"}]
assistant = autogen.AssistantAgent(
name="assistant",
llm_config={"config_list": config_list},
system_message="Write clean, well-documented Python code.",
)
user_proxy = autogen.UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
code_execution_config={"work_dir": "output", "use_docker": False},
max_consecutive_auto_reply=5,
)
user_proxy.initiate_chat(
assistant,
message="Write a function to calculate Fibonacci numbers, then test it with n=10."
)
Install: pip install pyautogen
Agno
What it is: Lightweight, Pythonic agent framework with minimal boilerplate and a large built-in tool ecosystem.
Best for: Quick prototypes. Clean code. When you want an agent running in under 10 lines.
from agno.agent import Agent
from agno.models.anthropic import Claude
from agno.tools.duckduckgo import DuckDuckGoTools
from agno.tools.file import FileTools
agent = Agent(
model=Claude(id="claude-3-5-sonnet-20241022"),
tools=[DuckDuckGoTools(), FileTools()],
description="Research assistant that finds information and saves summaries.",
markdown=True,
)
agent.print_response(
"Search for AI agent frameworks in 2025, summarise the top 3, and save to research.md",
stream=True,
)
Install: pip install agno
OpenAI Swarm
What it is: Lightweight educational framework showing how handoffs between agents work at a low level. Not intended for production.
Best for: Learning. Understanding the handoff pattern before using heavier frameworks.
from swarm import Swarm, Agent
client = Swarm()
writer = Agent(name="Writer", instructions="Write clear, structured content on any topic.")
def transfer_to_writer():
return writer
router = Agent(
name="Router",
instructions="Decide if the request needs research or writing, then transfer to the right agent.",
functions=[transfer_to_writer],
)
response = client.run(
agent=router,
messages=[{"role": "user", "content": "Write a short intro to AI agents"}]
)
print(response.messages[-1]["content"])
Smolagents (HuggingFace)
What it is: Code-first agents — instead of calling discrete tools, agents write and run Python code as their action. Works well with open-source models.
Best for: Data analysis, numerical tasks, workflows that map naturally to Python scripts. Open-source model use.
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel
agent = CodeAgent(
tools=[DuckDuckGoSearchTool()],
model=HfApiModel("Qwen/Qwen2.5-72B-Instruct"),
)
agent.run("Search for Python best practices in 2025 and list the top 5.")
Install: pip install smolagents
- LangGraph: most control, best for production — explicit graph-based flow
- CrewAI: fastest for role-based teams — researcher + writer + reviewer pattern
- AutoGen: conversation-based, great for code gen + execution workflows
- Agno: cleanest API, best for quick experiments and prototypes
- Swarm: educational only — learn handoffs, not for production
- Smolagents: code-first actions, best for open-source models and data tasks
- You are building a production pipeline that needs human approval before publishing — which framework would you use?
- You want to experiment quickly with a simple research agent in under 15 minutes — which framework?
- What is unique about how Smolagents differs from the other frameworks?
Observability with Langfuse
- Understand why observability is essential for agent systems
- Set up Langfuse and connect it to a LangGraph agent
- Know what to track in every agent run
Why observability matters
Without tracing, when your agent produces a wrong answer or costs $2 more than expected, you have no way to know what went wrong.
Agent Run Trace
|-- Input message: "Research AI agents and write a post"
|-- Step 1: LLM call
| |-- Input tokens: 312
| |-- Output tokens: 89
| +-- Model decided: call search tool
|-- Step 2: Tool call -> search("AI agents 2025")
| +-- Result: [3 search results]
|-- Step 3: LLM call
| |-- Input tokens: 890
| |-- Output tokens: 412
| +-- Model decided: write the draft, done
+-- Summary
|-- Total tokens: 1,703
|-- Total cost: $0.009
+-- Duration: 4.2 seconds
Setting up Langfuse
Self-host with Docker (see infra/docker/docker-compose.yml in this repo) or use langfuse.com (free tier):
pip install langfuse
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_HOST="http://localhost:3000"
Adding Langfuse to a LangGraph agent
from langfuse.callback import CallbackHandler
from langgraph.prebuilt import create_react_agent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
langfuse = CallbackHandler()
@tool
def search_web(query: str) -> str:
"""Search the web for information."""
return f"[Results for: {query}]"
agent = create_react_agent(
ChatAnthropic(model="claude-3-5-sonnet-20241022"),
tools=[search_web]
)
result = agent.invoke(
{"messages": [("human", "What is MCP?")]},
config={"callbacks": [langfuse]} # ← one line to add full tracing
)
Open Langfuse and you will see every LLM call, every tool call, token counts, costs, and duration — all from one line of code.
Adding a cost budget guard
from langchain_core.callbacks import BaseCallbackHandler
class CostGuard(BaseCallbackHandler):
def __init__(self, max_usd: float):
self.max_usd = max_usd
self.total = 0.0
def on_llm_end(self, response, **kwargs):
usage = response.llm_output.get("usage", {})
# Rough estimate for claude-3.5-sonnet
cost = (usage.get("input_tokens", 0) * 3 + usage.get("output_tokens", 0) * 15) / 1_000_000
self.total += cost
print(f" Cost so far: ${self.total:.4f}")
if self.total > self.max_usd:
raise RuntimeError(f"Budget exceeded: ${self.total:.4f} > ${self.max_usd}")
guard = CostGuard(max_usd=0.50)
# config={"callbacks": [langfuse, guard]}
What to track in every run
| What | Why |
|---|---|
| Input message | Understand what triggered the run |
| Each tool call + inputs | Debug wrong decisions |
| Each tool result | Spot tool failures |
| Token count per step | Find expensive steps |
| Total cost | Monitor spend |
| Final output | Evaluate quality |
| Errors | Catch and fix failures |
- Without observability, debugging agent failures is guesswork
- Langfuse traces every LLM call, tool call, token count, and cost
- One line adds full tracing:
config={"callbacks": [CallbackHandler()]} - Always set a cost guard on any long-running or production agent
Human-in-the-Loop
- Know which actions require human approval and which do not
- Implement LangGraph's interrupt pattern
- Build a trust level system for agent actions
Actions that need a human
Not every action should be automated. Irreversible or high-visibility actions need a human to approve first.
READ ONLY -> Auto-approve (no risk)
-----------------------------------------
search_web() [ok] automatic
read_file() [ok] automatic
query_database() [ok] automatic
LOW RISK -> Log but auto-approve
-----------------------------------------
write_file() [ok] automatic + logged
create_draft() [ok] automatic + logged
HIGH RISK -> Block until human approves
-----------------------------------------
send_email() [!!] human approval required
post_to_social() [!!] human approval required
delete_records() [!!] human approval required
deploy_code() [!!] human approval required
LangGraph interrupt pattern
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver
from langgraph.types import interrupt, Command
from typing import TypedDict
class State(TypedDict):
task: str
plan: str
approved: bool
result: str
def planner(state: State) -> State:
plan = f"1. Research '{state['task']}'\n2. Write a 300-word post\n3. Save to drafts/"
print(f"[Planner] Plan created:\n{plan}")
return {"plan": plan}
def approval_gate(state: State) -> State:
# interrupt() pauses the graph here and returns control to the caller
decision = interrupt({
"message": "Please review the plan and approve or reject:",
"plan": state["plan"],
})
return {"approved": decision.get("approved", False)}
def executor(state: State) -> State:
if not state["approved"]:
return {"result": "Cancelled by reviewer."}
return {"result": f"Completed: {state['plan']}"}
# Build with memory (required for interrupt)
memory = MemorySaver()
builder = StateGraph(State)
builder.add_node("planner", planner)
builder.add_node("approval", approval_gate)
builder.add_node("executor", executor)
builder.add_edge(START, "planner")
builder.add_edge("planner", "approval")
builder.add_edge("approval", "executor")
builder.add_edge("executor", END)
app = builder.compile(checkpointer=memory)
# ── Run 1: starts and pauses at approval_gate ──
thread = {"configurable": {"thread_id": "run-001"}}
for _ in app.stream(
{"task": "AI agent observability", "plan": "", "approved": False, "result": ""},
config=thread
):
pass
print("\n--- Agent paused. Human reviewing... ---")
# ── Run 2: resume with human decision ──
for event in app.stream(Command(resume={"approved": True}), config=thread):
if "executor" in event:
print("Result:", event["executor"]["result"])
- Classify tools by risk: read-only (auto), low-risk (log), high-risk (approve)
interrupt()pauses a LangGraph workflow and returns control to the caller- Resume with
Command(resume=data)— the agent continues from where it stopped - State is preserved between pause and resume via
MemorySaver
Hands-on Project: Multi-Agent Research Pipeline
- Build a complete 3-agent pipeline (Researcher → Writer → Publisher)
- Add Langfuse observability from the start
- Add a human approval gate before publishing
- Run the full pipeline end to end
Architecture
User gives a topic
|
v
+-------------------+
| Researcher Agent | <- searches web, compiles notes
+--------+----------+
| research notes
v
+-------------------+
| Writer Agent | <- writes blog draft from notes
+--------+----------+
| draft
v
+-------------------+
| [Approval Gate] | <- human reviews, approves/rejects
+--------+----------+
| approved = True/False
v
+-------------------+
| Publisher Agent | <- saves to file + database
+-------------------+
"""
research_pipeline.py — Complete multi-agent research and writing pipeline.
Install:
pip install langgraph langchain-anthropic langfuse duckduckgo-search
Run:
python research_pipeline.py
"""
import operator
from pathlib import Path
from typing import TypedDict, Annotated
from langchain_anthropic import ChatAnthropic
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.messages import HumanMessage, BaseMessage
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import StateGraph, START, END
from langgraph.types import interrupt, Command
from langfuse.callback import CallbackHandler
# ─── State ────────────────────────────────────────────────────────────────────
class State(TypedDict):
topic: str
research: str
draft: str
approved: bool
saved_to: str
messages: Annotated[list[BaseMessage], operator.add]
# ─── Setup ────────────────────────────────────────────────────────────────────
model = ChatAnthropic(model="claude-3-5-sonnet-20241022")
search = DuckDuckGoSearchRun()
langfuse = CallbackHandler()
OUTPUT_DIR = Path.home() / "ai-workspace" / "published"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
CALLBACKS = {"callbacks": [langfuse]}
# ─── Nodes ────────────────────────────────────────────────────────────────────
def researcher_node(state: State) -> State:
print(f"\n[Researcher] Researching: {state['topic']}")
raw = search.run(state["topic"])
response = model.invoke([
HumanMessage(content=
f"Synthesise these search results about '{state['topic']}' into structured notes:\n\n{raw}\n\n"
f"Cover: key concepts, recent developments, practical examples, limitations."
)
], config=CALLBACKS)
return {"research": response.content}
def writer_node(state: State) -> State:
print("[Writer] Writing draft...")
response = model.invoke([
HumanMessage(content=
f"Write a 500-word blog post about '{state['topic']}' using these notes:\n\n{state['research']}\n\n"
f"Requirements: specific title, hook opening, 3-4 body paragraphs, actionable conclusion. "
f"Tone: practical and honest, no hype words."
)
], config=CALLBACKS)
return {"draft": response.content}
def approval_node(state: State) -> State:
print("\n[Approval Gate] Pausing for human review...")
print("─" * 50)
print(state["draft"][:400], "\n...[truncated]")
print("─" * 50)
decision = interrupt({
"message": "Review the draft above. Approve to publish?",
"preview": state["draft"][:400],
})
return {"approved": decision.get("approved", False)}
def publisher_node(state: State) -> State:
slug = state["topic"].lower().replace(" ", "-")[:50]
path = OUTPUT_DIR / f"{slug}.md"
path.write_text(state["draft"], encoding="utf-8")
status = "published" if state["approved"] else "rejected"
print(f"[Publisher] Saved ({status}) → {path}")
return {"saved_to": str(path)}
# ─── Graph ────────────────────────────────────────────────────────────────────
def build_pipeline():
memory = MemorySaver()
g = StateGraph(State)
g.add_node("researcher", researcher_node)
g.add_node("writer", writer_node)
g.add_node("approval", approval_node)
g.add_node("publisher", publisher_node)
g.add_edge(START, "researcher")
g.add_edge("researcher", "writer")
g.add_edge("writer", "approval")
g.add_edge("approval", "publisher")
g.add_edge("publisher", END)
return g.compile(checkpointer=memory)
# ─── Run ──────────────────────────────────────────────────────────────────────
def run(topic: str):
app = build_pipeline()
thread = {"configurable": {"thread_id": f"run-{topic[:20].replace(' ', '-')}"}}
initial = {"topic": topic, "research": "", "draft": "", "approved": False, "saved_to": "", "messages": []}
print(f"\n{'='*50}")
print(f"Topic: {topic}")
print(f"{'='*50}")
for _ in app.stream(initial, config=thread):
pass # progress is printed inside nodes
# Simulate human review (set approved=False to reject)
print("\n>>> Human reviewing (auto-approving for demo)...")
for event in app.stream(Command(resume={"approved": True}), config=thread):
pass
print("\n>>> Done. Check Langfuse for full traces.")
if __name__ == "__main__":
run("How AI agents work and why they matter in 2025")
Running it
export ANTHROPIC_API_KEY="your-key"
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
python research_pipeline.py
After it runs, open Langfuse. You will see a trace for the full run, sub-spans for the researcher and writer model calls, token counts, and cost per step.
- A 3-agent pipeline (Researcher → Writer → Publisher) is built in ~100 lines with LangGraph
- Langfuse is added with one line and captures every step automatically
interrupt()pauses before publishing and waits for human inputCommand(resume=data)resumes the graph with the human's decision
Agent building checklist
- Tools have clear, specific descriptions
- Tools return structured data (dicts/lists), not free-form strings
- Tool errors return
{"error": "..."}, not exceptions - Every tool call is logged
- A token/cost budget is set
- Langfuse tracing is added from the start
- High-risk actions require human approval
- A maximum step limit is set
Quick reference
| Term | Meaning |
|---|---|
| Agent | LLM + tools + memory + loop — autonomously completes goals |
| ReAct | Reason + Act — the standard agent loop pattern |
| Tool calling | API mechanism for LLMs to request function execution |
| Orchestrator | Agent that delegates tasks to specialist subagents |
| Human-in-the-loop | Required human approval before consequential actions |
| Observability | Tracing every step, token, and cost of agent runs |
| LangGraph | Graph-based framework for stateful agent workflows |
| Langfuse | Open-source observability platform for LLM applications |
| Interrupt | LangGraph mechanism to pause a graph for human input |
| State | Shared data that flows through a LangGraph graph |
Questions? Email hello@pranavsrivastava.com