Attention & Transformers

Almost every AI you have heard of — ChatGPT, Claude, Gemini, image generators, translation, even the tool folding proteins in a lab — runs on one architecture: the transformer. And the transformer is built from one idea, repeated and stacked: attention. This course does not wave its hands at that idea. It builds it from the ground up — the actual arithmetic, worked by hand with numbers small enough to check yourself — until you could, if you wanted to, explain the whole thing to a curious 14-year-old without lying to them once.

Before attention existed, the standard way to have a neural network read a sentence was a recurrent neural network, or RNN — covered in the Deep Learning course. Picture it like someone reading a book one word at a time through a cardboard tube, jotting a short note in the margin after every word to remind themselves what mattered so far, then covering that word up and moving to the next. The note is all they carry forward. Everything not captured in the note is gone.

That "note" is called the hidden state, and it has a fixed size — the same number of numbers, no matter how long the sentence gets. Try to picture what that means: a two-word sentence and a two-hundred-word paragraph get squeezed into the exact same amount of memory. Something has to get dropped, and it's usually whatever came first.

Here's where that actually bites. Read this sentence and answer instantly, without thinking about it:

"The trophy did not fit in the brown suitcase because it was too big."

What does "it" refer to? You knew before you finished reading — the trophy, obviously; a suitcase being "too big" wouldn't stop something fitting inside it. Now read this one:

"The trophy did not fit in the brown suitcase because it was too small."

Change one word — "big" to "small" — and "it" now means the suitcase. You didn't even notice yourself doing this. Your brain silently reached all the way back across ten words, picked out exactly the right noun, and updated its answer the instant "small" appeared. That's not a party trick. Linguists call this exact kind of sentence a Winograd schema, and it's used as a genuine test of whether a system understands language or is just pattern-matching, because getting it right requires holding the whole sentence in mind at once, not reading it through a straw.

Before any maths, let's get the feeling of attention right, because the feeling is the whole idea — the arithmetic in the next chapter is just a way of making the feeling precise enough for a computer to do it.

You already do this constantly. At a noisy party, dozens of conversations are hitting your ears at once, but the moment someone across the room says your name, you hear it instantly — your brain was quietly monitoring everything, ready to snap focus onto whatever mattered. Psychologists call this the cocktail party effect, and it's a genuinely good description of what attention does inside a model: it isn't reading one thing at a time and forgetting the rest. It's looking at everything at once, and deciding, for whatever it's currently focused on, which of all the other things actually matter — and by how much.

Or think about skimming a long legal contract. You hit the phrase "the aforementioned party" and your eyes don't re-read the whole document from the top — they flick back to the exact clause that defined who that was, ignore everything else on the page, and return. That flick — precise, selective, instant — is attention. Not reading everything with equal care. Reading everything, but unevenly, on purpose, based on what's relevant right now.

Here is the part almost every "attention explained simply" article skips, and it's the part that actually matters. The cocktail-party feeling is nice, but how does a model actually compute "focus on this word 70% and that one 20%"? This chapter builds it from scratch, with a library-desk analogy and numbers small enough to check by hand.

Imagine a librarian's front desk. Every word in the sentence, as it's processed, does three things:

• It writes a Query — a short description, as a list of numbers, of what it's looking for to complete its own meaning.
• It writes a Key — a short tag describing what kind of information it offers to anyone searching.
• It carries a Value — its actual content, what gets handed over if it's chosen.

Query, Key and Value are the same idea a search engine uses: your search box is the Query, the index tags on every webpage are the Keys, and the actual page content is the Value. Attention is a search, run separately for every single word, against every other word in the sentence, all at once.

Walk through what just happened, because every one of those five steps has a name worth knowing: the model computed a score for how well the Query matched each Key (a dot product — just "multiply and add," a way of measuring how alike two lists of numbers are); it ran those scores through softmax, which is a clean mathematical way of turning any list of numbers into honest percentages that add up to exactly 1; and it used those percentages to blend the Values into one new, context-aware meaning. That entire five-step recipe — Query, Key, Score, Softmax, weighted Value — is attention. Everything else in this course is either preparing inputs for that recipe or stacking many copies of it together.

import numpy as np

def softmax(x):
    x = x - np.max(x)  # for numerical stability
    e = np.exp(x)
    return e / e.sum()

def self_attention(X, Wq, Wk, Wv):
    """
    X:  (seq_len, d_model) — one embedding row per word
    Wq, Wk, Wv: (d_model, d_k) — learned projection matrices
    Returns: (seq_len, d_k) — one new, context-aware vector per word
    """
    Q = X @ Wq                          # every word's Query
    K = X @ Wk                          # every word's Key
    V = X @ Wv                          # every word's Value

    d_k = K.shape[-1]
    scores = (Q @ K.T) / np.sqrt(d_k)   # every Query vs. every Key
    weights = np.apply_along_axis(softmax, 1, scores)  # rows sum to 1

    return weights @ V                  # weighted blend of Values

# A toy sentence with 4 words, each an 8-number embedding
X = np.random.randn(4, 8)
Wq, Wk, Wv = (np.random.randn(8, 8) for _ in range(3))

output = self_attention(X, Wq, Wk, Wv)
print(output.shape)  # (4, 8) — one new, context-aware vector per word

Theory earns its keep once you can actually watch it happen. Here is the Winograd sentence from Chapter 1, with attention weights laid out exactly the way a real trained model's would look — pick a word, see where its attention goes, and watch what happens when one word in the sentence changes.

Sit with the second example for a second, because it's the whole argument in one click. Nothing about where "it" sits in the sentence changed — same position, ninth word, every time. The only thing that changed was meaning — "big" swapped for "small" — and the model's attention followed the meaning, not the position. That is the entire case for attention over the old straw-reading approach, made concrete: it isn't tracking distance. It's tracking relevance.

One attention "search" is already powerful — but a sentence has many kinds of relationships happening simultaneously. "Quick" describes "fox." "Fox" is the subject of "jumps." "Fox" and "dog" are quietly linked as the two animals in the scene. Asking one single Query/Key/Value setup to capture all of that at once is asking a lot from one spotlight.

So transformers don't use one. They run several attention "heads" in parallel — each with its own learned Query, Key and Value matrices — so each head is free to specialise in a different kind of relationship, entirely on its own, during training.

Nobody tells Head 1 "you're in charge of adjectives" or Head 4 "you handle animal pairs." Those specialisations emerge from training, purely because splitting the work that way happens to make the model better at predicting language. Researchers going back into trained models afterward and finding heads that cleanly track syntax, or coreference, or position, is one of the genuinely delightful discoveries in this whole field — a bit like performing an autopsy and finding the patient had quietly organised their own internal departments.

Here's a gap in everything we've built so far, and it's a genuinely important one: nothing about Query, Key, Value or softmax mentions order. Attention compares every word to every other word as an unordered set — as far as the maths in Chapter 3 is concerned, "the dog bit the man" and "the man bit the dog" contain exactly the same words attending to exactly the same words. Obviously those two sentences mean opposite things. Something has to tell the model which word came first.

The fix, called positional encoding, is one of the more elegant ideas in the whole architecture. Before attention ever runs, every position in the sentence gets its own unique numeric fingerprint, generated from waves — and it's added directly onto that word's embedding.

The clock analogy is worth sitting with, because it explains why waves specifically. Picture a clock with several hands moving at very different speeds. The fastest hand distinguishes this second from the last one — perfect for telling neighbouring positions apart, useless for anything far away, since it loops back round constantly. The slowest hand barely moves at all over a short span — useless for nearby positions, perfect for telling position 3 apart from position 300. Positional encoding uses a whole family of these hands, each ticking at a different frequency, and together they give every single position in a sentence — however long — its own fingerprint that no other position shares, while also making nearby positions look naturally similar to each other. That second property matters: the model doesn't just learn "this is position 9." It learns "this position is close to position 8 and far from position 40," which turns out to be exactly the kind of information attention can put to good use.

Time to assemble every piece into the actual architecture. A single transformer block is smaller than it sounds once you've built each part — it's multi-head attention, plus a couple of standard neural-network habits that make deep networks trainable at all.

Two pieces here weren't covered yet, and both exist purely to make very deep networks actually trainable:

The residual connection ("Add") is simple and important: alongside the attention layer's output, the block also adds back the original input, unchanged. Think of it as a highway running straight through the block, with attention as an off-ramp that adds a correction rather than replacing the signal entirely. Without this, stacking 96 layers tends to make training unstable — small errors compound layer after layer until the signal is unrecognisable. The highway means even if one layer contributes almost nothing useful, information can still flow cleanly past it.

Layer normalisation ("Normalise") rescales the numbers flowing through the network so they stay in a sane, consistent range at every layer — roughly, it keeps the "volume" from drifting too loud or too quiet as data passes through dozens of stacked blocks, which again keeps training stable.

The feed-forward network is the one genuinely new piece: after attention has gathered context from other words, each word passes through its own small, ordinary neural network — the same one, reused for every word — giving the model extra capacity to process what attention just found, on a word-by-word basis, before passing everything to the next block.

There were plenty of clever sequence-processing ideas before and around the transformer. Why did this one take over so completely? Because it hit a genuinely rare combination: it was better at understanding long-range meaning (any word can attend directly to any other, no matter the distance — the straw is gone entirely) and, separately, cheaper to scale.

That second part is easy to undervalue, so it's worth spelling out. An RNN has to process word 1, then word 2, then word 3 — strictly in order, because each step needs the previous one's result. That's a hard ceiling on how much you can speed things up with more hardware. Attention has no such requirement: every word's Query, Key and Value can be computed simultaneously, and every attention score can be computed in parallel too. That maps almost perfectly onto how GPUs work — thousands of small calculations happening at once — which meant transformers could be trained on vastly more text, vastly faster, for a given amount of hardware.

One more piece completes the picture, and it explains something that might otherwise seem strange: if attention lets every word see every other word at once, why does ChatGPT still visibly type its answer out one word at a time, left to right?

The transformer architecture actually comes in three shapes, depending on the job:

• An encoder reads an entire input and lets every word attend freely to every other word, including ones that come after it — good for understanding a full piece of text at once (translation input, classification, search).
• A decoder generates text one step at a time, and uses causal (masked) attention: when producing word 12, it's mathematically forbidden from attending to words 13 onward, because they don't exist yet. This is precisely why ChatGPT, Claude, and essentially every modern chat model write left to right — it isn't a UI choice, it's baked into how the model is allowed to attend.
• An encoder-decoder pairs both: an encoder fully understands the input (say, a French sentence), and a decoder generates the output (the English translation) one word at a time, attending both to its own previous words and, separately, to the encoder's full understanding of the input — this second kind of attention, pointing at a different piece of text entirely, is called cross-attention.

The remarkable postscript to all of this: attention was invented for language, and it turned out not to care what kind of data it was looking at.