Embeddings, in Depth: Turning Words into Numbers

There is one idea hiding underneath ChatGPT, Claude, Google search, Spotify recommendations, and almost every "smart" thing a computer does today. It is not complicated — an eight-year-old can get it in a minute — but it is so powerful it took decades to fully exploit. The idea is this: turn words into numbers, arranged so that the numbers carry the meaning. Those numbers are called embeddings, and this course goes from the picture a child can grasp all the way to the billions of numbers inside a frontier model.

Some different words mean nearly the same thing — cat, kitten, feline — and so do car, vehicle, automobile. A computer that understood meaning should treat the words in each group as basically the same thing. Press the button and watch it do exactly that:

Sorted by letters, "cat" lands next to "car" — useless. Sorted by meaning, words that mean the same thing snap into the same place. That "same place" is the whole idea. Do it for every word at once and you get a giant map where similar meanings are neighbours:

The computer was never handed a dictionary. It read a mountain of text, noticed which words get used in similar ways, and placed them close together. That map of meaning is what embeddings are.

On paper we draw the map flat, with two directions: left-right and up-down. That is 2 dimensions. Add depth — a direction into the page — and you have 3 dimensions, a real space you can spin. Grab this one and give it a whirl; tap a word to see its nearest neighbours:

Here is the mind-bender: a real embedding does not use 2 or 3 directions. It uses hundreds or thousands of them at once. You cannot picture a 768-dimensional space — nobody can — but the maths works exactly the same as this 3D toy: every word is a point, and "similar meaning" means "close together," just in far more directions. Each extra dimension is another subtle question the model can use to tell meanings apart ("is it alive?", "is it hot?", "is it formal?" — though the real ones aren't labelled so neatly).

Because every word is now a point, the relationships between meanings become directions you can add and subtract. The famous example: take "king", apply the same step that turns "man" into "woman", and you land on "queen." Build your own and watch the two arrows stay parallel:

This result — man is to woman as king is to queen, falling out of pure geometry — was the moment, back in 2013, that convinced researchers embeddings had captured real meaning and not just word-counting. It came from a Google system called word2vec (Mikolov et al., 2013), and it is still the clearest proof that "distance means similarity, direction means relationship."

Nobody places the words by hand. The machine learns the whole map from raw text, using one deceptively simple principle, stated by the linguist J.R. Firth back in 1957: "You shall know a word by the company it keeps."

The training trick is a fill-in-the-blank game, played billions of times:

  "The ___ chased the mouse."     → the model guesses: cat? dog? kitten?
        │
        ▼
  It was "cat". Nudge the numbers so 'cat' fits this context better.
        │  (repeat over billions of sentences)
        ▼
  Words that appear in similar blanks drift close together — automatically.

Because "cat" and "kitten" show up in the same kinds of sentences, the nudges keep pushing them together, and they end up neighbours — no human ever told the machine they were related. Do this over billions of sentences and the entire map of meaning organises itself. That is self-supervised learning, and it is the engine behind every large model.

A large language model is, at its front door, a giant embedding table. Here is the journey your words take:

  "hello there"  ──split──►  tokens: [hello] [ there]
                                │  look each up in the embedding table
                                ▼
                    each token → a vector of thousands of numbers
                                │  fed through dozens of transformer layers
                                ▼
                    the model predicts the next token — and writes

Two numbers make the scale real. A model has a vocabulary of tokens (word-pieces) — tens of thousands of them — and each token's embedding is a vector whose length is the model's hidden dimension. For GPT-3, that was a vocabulary of about 50,000 tokens and a hidden dimension of 12,288. Multiply them and the input embedding table alone is roughly 617 million numbers — before the model does a single bit of thinking.

That table is only the entrance. Above it sit dozens of transformer layers (from the 2017 paper "Attention Is All You Need"), and all of those numbers together are the model's parameters — the knobs learned during training. Every version you have heard of is mostly a story about how many of those knobs there are.

Now the facts, because the growth is genuinely hard to believe. In a few years, models went from millions of parameters to hundreds of billions, trained on more text than any human could read in a thousand lifetimes.

A few facts to sit with:

• Compute. Training GPT-3 took an estimated 3.14 × 10²³ operations — the equivalent of running a fast laptop for thousands of years, compressed onto thousands of GPUs running for weeks.
• Cost. Independent estimates put GPT-3's training compute around $4.6 million; OpenAI's CEO said GPT-4 cost "more than $100 million"; the 2024 Stanford AI Index estimated frontier training runs (GPT-4, Gemini Ultra) in the $78M–$191M range. Frontier costs are still climbing.
• Data. GPT-3 read about 300 billion tokens; modern models train on 10–15 trillion. The 2022 Chinchilla result (Hoffmann et al., DeepMind) showed that, for a given compute budget, you often want more data per parameter — roughly 20 tokens for every parameter — reshaping how everyone trains.

Sources: Mikolov et al., “Efficient Estimation of Word Representations,” 2013 (word2vec); J.R. Firth (1957), the distributional hypothesis; Vaswani et al., “Attention Is All You Need,” 2017; Brown et al., “Language Models are Few-Shot Learners,” 2020 (GPT-3: 175B params, 12,288 hidden dim); Hoffmann et al., “Training Compute-Optimal LLMs” (Chinchilla), 2022; the Stanford AI Index 2024 (training-cost estimates). Parameter/cost figures for GPT-4 and later are estimates; exact numbers are undisclosed. This course underpins Folders vs Vectors and RAG in Production.