Attention & Transformers

Almost every AI you have heard of — ChatGPT, Claude, image generators, translation — runs on one architecture: the transformer. And the transformer is built on one elegant idea: attention. This course explains that idea so clearly you will see why it changed everything.

Recall the RNN from the Deep Learning course: it reads a sentence one word at a time, carrying a small memory forward. The trouble is that memory fades. By the end of a long paragraph, the beginning is a blur.

Consider: "The trophy did not fit in the brown suitcase because it was too big." What does "it" refer to — the trophy or the suitcase? You know instantly: the trophy. But an RNN, having shuffled word-by-word, often loses that thread. It read the sentence through a straw, remembering only a little at a time.

Attention throws out the straw. Instead of reading word by word and forgetting, the model looks at the whole sentence at once and, for each word, decides which other words matter most to understanding it.

When the model processes "it" in our sentence, attention lets it glance back and ask: which earlier word am I really about? It weighs every word and decides "trophy" matters most. It has, quite literally, learned where to pay attention.

  The  trophy  did  not  fit  ...  because   it   was big
        ▲                                    │
        └──────────── strong attention ──────┘
        (weaker attention to every other word too)

A human analogy: reading a contract, when you hit "the aforementioned party," your eyes flick back to find who that was. You do not re-read the whole document — you attend to the relevant part. Attention gives a model that same ability, for every word, all at once.

A transformer is what you get when you stack many attention layers together. Each layer lets every word gather context from every other word; the next layer builds richer understanding on top of that; and so on, deepening meaning at each level — just like the layer hierarchy from the Deep Learning course.

It has two superpowers:

1. It captures long-range meaning — any word can attend directly to any other, no matter how far apart. The straw is gone.
2. It is fast to train — because it looks at all words at the same time (in parallel) rather than one after another, it makes full use of modern hardware. RNNs were stuck going one step at a time.

There were many clever architectures over the years. Why did the transformer take over so completely? Because it hit a rare sweet spot: it was better at understanding (long-range attention) and cheaper to scale (parallel training) at the same time. Usually you trade one for the other. The transformer gave both.

That combination meant: feed it more text and more computing power, and it kept getting better — predictably. This "just add scale" property is why a single architecture now underlies chatbots, code assistants, translation, image generation, and more. They are, under the hood, the same machine pointed at different data.