# How Chatbots Think

> How large language models such as ChatGPT, Claude and Gemini actually work: tokens, embeddings, next-word prediction, training by gradient descent, attention and the Transformer, scale, human feedback, hallucinations, tools and reasoning, with live demos you can play with.

Interactive film: https://www.planetpast.com/stories/how-llms-work/  
Publisher: PlanetPast (https://www.planetpast.com/) · Published 2026-10-13 · Technology, Math

## Prologue

### How chatbots think
*October 2026*

You type a question. A few seconds later, fluent text appears. What actually happens in between?

On the seventh of October 2026, OpenAI started rolling out its new GPT-6 model, with a more visual interface, to ChatGPT users worldwide. Chatbots like ChatGPT, Claude and Gemini are now part of daily life. ChatGPT alone reported nine hundred million weekly users in early 2026. They can sound like a person. But inside there is no little person: there is a very large mathematical function, built from numbers. Its job is simple to state: read some text, and guess what comes next. This is how that simple job grows into something that can write, explain and argue.

Prologue · Why this matters now

#### A question goes in, an answer comes out

On 7 October 2026 OpenAI started rolling out GPT-6 to ChatGPT users worldwide, the latest release in a fast-moving field that includes companies such as OpenAI, Anthropic, Google and Meta, and that took off when ChatGPT launched on 30 November 2022. All of these systems are **large language models** (LLMs): neural networks trained on enormous amounts of text to generate language. This film explains how they work, one idea at a time, without assuming any maths.

- **ChatGPT launched**: 30 Nov 2022 100 million users in two months

- **Weekly users**: 900 million ChatGPT, reported February 2026

- **The core trick**: Predict the next piece of text, again and again Everything else in this film is about how to make that guess good

The swirling galaxy on the left is a picture of a model’s vocabulary: each little square stands for a piece of text. Blue streams carry your prompt in; gold streams carry the answer out. It is an illustration, not a measurement.

How to use: pick a chapter below the scene or press ← → to change step · ▶ plays the step · drag to rotate, scroll or pinch to zoom · “Play the film” runs every step full-screen. Several steps have live demos in this panel.

Sources: [TechCrunch, 7 Oct 2026](https://techcrunch.com/2026/10/07/chatgpt-is-getting-a-lot-more-visual-with-the-launch-of-a-new-interface/); [Wikipedia: ChatGPT](https://en.wikipedia.org/wiki/ChatGPT); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Chapter I · Tokens

### Cutting text into pieces
*Chapter I · Tokens*

A computer cannot read letters. Everything has to become numbers first.

A language model never sees letters. The first thing that happens to your message is that it is chopped into pieces called tokens. Common words get a token of their own. Rare words are built from several smaller pieces. Each piece has an ID number in a fixed vocabulary. Then every ID is swapped for a long list of numbers, a vector. From here on, the model only ever works with these vectors. In English, one token is about three quarters of a word. Try typing your own text in the panel.

Chapter I · Step 1

#### Text becomes tokens

Before a model can do anything, the text is split into **tokens**: whole words, pieces of words, single characters or punctuation. Most modern tokenizers use a method called **byte-pair encoding** (BPE), first described by Philip Gage in 1994: start from single characters and repeatedly merge the pair that appears together most often, until the vocabulary reaches a chosen size.

Try it: a tokenizer*simplified*
A simplified byte-pair encoder written for this page. Its vocabulary of about 1,100 pieces was learned from the opening chapters of *Alice’s Adventures in Wonderland* plus a few Thai paragraphs, so the ids are this demo’s own. Real tokenizers learn 50,000–200,000 pieces from far more text. Pieces it never learned fall back to raw bytes, shown like ‹E0›.

- **Rule of thumb (English)**: 1 token ≈ 4 chars 100 tokens ≈ 75 words; other languages differ

- **Vocabulary of GPT-4**: 100,258 100,000 BPE pieces + 258 special tokens

The ratio of tokens to words differs from language to language. Thai, for example, is written without spaces between words; type some Thai into the box to see how this small tokenizer copes.

Sources: [Wikipedia: Byte-pair encoding](https://en.wikipedia.org/wiki/Byte-pair_encoding); [OpenAI Help Center: What are tokens](https://help.openai.com/en/articles/4936856-what-are-tokens); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Chapter II · Meaning as direction

### Meaning as direction
*Chapter II · 2013*

Words that are used in similar ways end up close together, and relationships become arrows.

Each token becomes a point in a space with hundreds or thousands of dimensions. We can only draw three, so this picture is a simplified sketch. The positions are learned from text. Words that appear in similar company, like cat and dog, drift close together. Even more surprising: relationships become directions. The step from man to woman is roughly the same as the step from king to queen. In 2013 researchers showed that king, minus man, plus woman, lands very close to queen. Meaning had become geometry.

Chapter II · Step 2 · 2013

#### Tokens become vectors

Each token ID is replaced by an **embedding**: a list of numbers that places it as a point in a high-dimensional space. Nobody sets these numbers by hand; they are learned, because words used in similar contexts get similar vectors. In 1957 the linguist J. R. Firth put the idea in a phrase: a word is characterised by the company it keeps.

Try it: word arithmetic*hand-placed, illustrative*

- **word2vec (Google)**: 2013 Mikolov and colleagues; vectors of ~100–1,000 numbers

- **GPT-3 vectors**: 12,288 numbers per token inside the model

- **The famous result**: “King − Man + Woman” is very close to “Queen” Mikolov, Yih and Zweig, 2013

The 3D points here were placed by hand to show the idea; real embeddings have far more dimensions and are messier. Embeddings also pick up biases in their training text: a 2016 study found analogies such as “man is to computer programmer as woman is to homemaker”.

Sources: [Mikolov, Yih & Zweig, NAACL-HLT 2013](https://aclanthology.org/N13-1090.pdf); [Wikipedia: Word2vec](https://en.wikipedia.org/wiki/Word2vec); [Wikipedia: Word embedding](https://en.wikipedia.org/wiki/Word_embedding); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3)

## Chapter III · The next word

### Guessing the next word
*Chapter III · Prediction*

The model never writes a whole answer at once. It picks one token, adds it to the text, and guesses again.

Given everything written so far, the model outputs a score for every token in its vocabulary. A function called softmax turns those scores into probabilities that add up to one hundred per cent. Then one token is drawn, like a weighted dice roll. It is added to the text, and the whole process repeats. A setting called temperature controls the dice. Low temperature: safe, predictable choices. High temperature: surprising ones, and eventually nonsense.

Chapter III · Step 3

#### Predicting the next token

A language model is, at heart, a next-token predictor. For the text so far, it produces a probability for every token in its vocabulary. A token is picked, appended, and the model runs again with the longer text. Long answers are built this way, one token at a time.

Try it: a tiny next-word predictor*trigram toy*
Temperature
This toy only looks at the last two words, and it learned from about 6,000 words of *Alice’s Adventures in Wonderland* (chapters I–III, public domain), so it writes Carroll-flavoured English whatever language you read in. A real model looks at thousands of tokens and learned from trillions.

- **Scores.** The network outputs one number per vocabulary token.

- **Softmax.** Exponentiate and normalise so they sum to 1 (in physics the same formula is the Boltzmann distribution, 1868).

- **Temperature.** Divide the scores by T first: low T sharpens the distribution towards the top choice; high T flattens it.

- **Sample** one token, append it, repeat.

Sources: [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model); [Wikipedia: Softmax function](https://en.wikipedia.org/wiki/Softmax_function); text: [Project Gutenberg eBook #11](https://www.gutenberg.org/ebooks/11)

## Chapter IV · Learning

### Billions of dials
*Chapter IV · Learning*

Where do the probabilities come from? From a network of numbers that starts out random and is tuned, nudge by nudge.

The scores come from a neural network: layers of simple units joined by connections. Each connection has a weight, a number like a dial. Training starts with random dials, so the first guesses are nonsense. The model reads real text and guesses the next word. The guess is compared with the word that really came next. The difference is the error, or loss. An algorithm called backpropagation works out how each dial should turn to make the error a little smaller. Then every dial is nudged. Repeat, trillions of times.

Chapter IV · Step 4

#### A network of adjustable numbers

The model is a neural network: numbers flow forward through layers, multiplied by **weights** and passed through simple functions. These weights (and a few other numbers) are the model’s **parameters**. Training is self-supervised: the “right answer” is simply the next token of real text, so no human has to label anything.

- **Forward:** the network reads text and outputs probabilities for the next token (blue wave).

- **Compare:** the loss measures how much probability it gave to the token that actually came next.

- **Backward:** backpropagation computes, for every weight, which way to turn it to reduce the loss (orange wave).

- **Update:** nudge every weight a tiny step that way. Watch “get” become more likely in the scene.

- **This toy network**: 237 parameters (weights + biases)

- **GPT-3 (2020)**: 175 billion parameters

- **Backpropagation**: Linnainmaa 1970 · Werbos 1982 · Rumelhart, Hinton & Williams 1986 the 1986 *Nature* paper made it widely known

Sources: [Wikipedia: Backpropagation](https://en.wikipedia.org/wiki/Backpropagation); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3)

### Walking downhill in fog
*Chapter IV · Gradient descent*

Imagine every possible setting of the dials as a landscape. The height is the error. Training is a long walk downhill.

Picture the loss as a landscape. Every point is one setting of the model’s weights, and the height is how wrong the model is. You cannot see the whole landscape, only the slope right under your feet. So you take a step downhill, and look again. This is gradient descent, an idea attributed to the mathematician Augustin-Louis Cauchy in 1847. Real models walk in a space with billions of directions, not two. Try different step sizes in the panel: too small is slow, too large overshoots.

Chapter IV · Step 5 · 1847 → today

#### Gradient descent: following the slope

The **gradient** tells you which direction is steepest uphill. Step the other way and the loss goes down. Repeat millions of times on batches of text and the loss falls steadily. The size of each step is the **learning rate**.

Try it: step size*2D toy landscape*

- **Idea first suggested**: 1847 Augustin-Louis Cauchy

- **Cost of learning one token**: ≈ 6 FLOPs per parameter (rule of thumb)

- **What GPT-3 read**: 60% filtered web crawl · 22% WebText2 · 16% books · 3% Wikipedia shares of GPT-3’s training mix, 2020

The green glow marks the lowest point of this toy landscape; the pink one is a shallow dip where a walker can get stuck. In very high dimensions, training usually finds good-enough valleys rather than the single best one.

Sources: [Wikipedia: Gradient descent](https://en.wikipedia.org/wiki/Gradient_descent); [Wikipedia: Neural scaling law](https://en.wikipedia.org/wiki/Neural_scaling_law); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3)

## Chapter V · Attention

### Attention
*Chapter V · 12 June 2017*

Which earlier words matter for this one? A Transformer asks that question for every word, all at once.

Words change meaning with context. In this sentence, what does “it” refer to: the animal, or the street? Self-attention lets every token look back at every earlier token and decide how much each one matters right now. When the last word is “tired”, the strongest link points to the animal. Change one word to “wide”, and the link swings to the street. Each token sends out a query, compares it with every other token’s key, and gathers their values in proportion to the match. A model runs many of these attention heads side by side. Some track meaning, others simply track the word just before.

Chapter V · Step 6 · 2017

#### Self-attention

In June 2017 eight researchers at Google published “Attention Is All You Need”, introducing the **Transformer**. Its key part, **self-attention**, lets each token gather information from the tokens before it, weighting each by how relevant it is.

Try it: pick a word*illustrative weights*
Tap a word (here or in the scene) to see how much it “looks at” each earlier word. These weights were written by hand to show the idea; a real model learns its own, and different heads in the same layer look for different things.

- Each token’s vector is turned into a **query** (what am I looking for?), a **key** (what do I contain?) and a **value** (what do I pass on?).

- Query · key gives a match score for every earlier token; the scores are scaled and passed through softmax, so they add up to 100%.

- The token’s new vector is the weighted mix of the values. In chat models a mask stops tokens from looking ahead.

Sources: [Wikipedia: Attention Is All You Need](https://en.wikipedia.org/wiki/Attention_Is_All_You_Need); [Wikipedia: Transformer (deep learning)](https://en.wikipedia.org/wiki/Transformer_(deep_learning))

### A tower of layers
*Chapter V · The Transformer*

Attention, then a small network for each token, then again. And again. Ninety-six times in GPT-3.

A Transformer layer has two parts: attention, where tokens share information, and a feed-forward network that processes each token on its own. Layers are stacked. Each one refines every token’s vector a little more, folding in context from the rest of the text. Because all tokens are processed at the same time, the work spreads well across many chips. That is a big reason the design took over. At the top, the final vector of the last token is turned into scores for the next token. And the cycle begins again.

Chapter V · Step 7

#### The Transformer, stacked

Earlier language networks (recurrent networks, such as LSTMs) read one word at a time. The Transformer handles all positions of a passage in parallel, which suits graphics processors (GPUs). The original 2017 model, built for translation, had about 100 million parameters; its larger version trained for 3.5 days on eight GPUs. Chat models such as the GPT series keep only the “decoder” half.

- **GPT-3 layers**: 96 each with 96 attention heads

- **Vector width**: 12,288 numbers per token

- **Context (GPT-3)**: 2,048 tokens it could see at once

- **The 2017 paper**: 250,000+ citations as of 2026

The scene shows 12 layers and 6 tokens; GPT-3 has 96 layers and handles up to 2,048 tokens. The title of the paper plays on the Beatles song “All You Need Is Love”.

Sources: [Wikipedia: Attention Is All You Need](https://en.wikipedia.org/wiki/Attention_Is_All_You_Need); [Wikipedia: Transformer (deep learning)](https://en.wikipedia.org/wiki/Transformer_(deep_learning)); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3)

## Chapter VI · Scale

### Bigger, and then bigger
*Chapter VI · Scale*

More parameters, more text, more computing power. For years, simply scaling up kept making models better.

In January 2020, researchers reported that the error of language models falls smoothly and predictably as models, data and computing power grow. Within a few years, models grew from millions of parameters to hundreds of billions. Each step on this tower is ten times taller than the last. In 2022, DeepMind’s Chinchilla showed that many models had been too big for the data they read. With a quarter of the size and four times the data, it beat a much larger model. All of this needs enormous computing power. One estimate put GPT-3’s training at three hundred and fifty-five years on a single graphics chip; in practice it was spread across many chips working in parallel.

Chapter VI · Step 8

#### Scaling laws

A 2020 study by Jared Kaplan and colleagues found that a language model’s loss “scales as a power-law with model size, dataset size, and the amount of compute used for training”. In plain words: make each of them bigger and the error drops by a predictable amount. Larger models also showed new abilities; whether those appear suddenly or smoothly is still debated.

- **GPT-2 (2019)**: 1.5 billion parameters

- **GPT-3 (2020)**: 175 billion parameters

- **Chinchilla (2022)**: 70 B · 1.4 T parameters · training tokens (≈ 20 tokens per parameter)

- **Llama 3.1 (2024)**: 405 B · ~15 T parameters · training tokens

- **GPT-3 on one GPU**: ~355 years estimate (Lambda Labs, 2020); real training ran on many GPUs in parallel

- **One Nvidia H100 (2022)**: 80 B · 700 W transistors · peak power

Not every developer publishes model sizes: OpenAI did not disclose GPT-4’s (2023), though its chief executive said training it cost more than $100 million. Many recent models also use a “mixture of experts”, in which each token passes through only a few of many sub-networks.

Sources: [Kaplan et al., Scaling Laws for Neural Language Models (2020)](https://arxiv.org/abs/2001.08361); [Wikipedia: Neural scaling law](https://en.wikipedia.org/wiki/Neural_scaling_law); [Wikipedia: Chinchilla](https://en.wikipedia.org/wiki/Chinchilla_(language_model)); [Wikipedia: Llama](https://en.wikipedia.org/wiki/Llama_(language_model)); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3); [Wikipedia: GPT-4](https://en.wikipedia.org/wiki/GPT-4); [Wikipedia: Hopper / H100](https://en.wikipedia.org/wiki/Nvidia_H100); [Wikipedia: Mixture of experts](https://en.wikipedia.org/wiki/Mixture_of_experts)

## Chapter VII · From text machine to assistant

### From text machine to assistant
*Chapter VII · 2022*

A model trained only to continue text is not yet a helpful assistant. Two more rounds of training shape it.

A freshly trained base model only continues text. Ask it to write a poem, and it may just list more requests that look like yours. So it is fine-tuned on many examples of instructions paired with good answers. This is called instruction tuning. Then people compare pairs of answers and pick the better one. A second model, the reward model, learns to predict those choices. Finally, reinforcement learning nudges the assistant towards answers the reward model scores highly, while keeping it close to what it learned before.

Chapter VII · Step 9 · 2022

#### Instruction tuning and human feedback

Pre-training produces a **base model** that knows a lot about text but simply continues whatever it is given. Turning it into an assistant takes further training, which OpenAI described for InstructGPT in 2022, shortly before ChatGPT.

- **Pre-training:** predict the next token on a huge text collection (chapters III–VI).

- **Instruction tuning:** supervised training on prompts paired with good, human-written responses.

- **Reinforcement learning from human feedback (RLHF):** people rank answers; a reward model learns those preferences; the assistant is optimised against it, with a penalty for drifting too far from the tuned model.

- **Who uses it**: ChatGPT, Gemini and Claude are among the models trained with RLHF Variants include direct preference optimisation (DPO) and AI feedback guided by written principles (“Constitutional AI”)

The method has known weak spots: human ratings are costly and can be inconsistent or biased, and a model can learn to please the reward model, for example by sounding confident, rather than to be right.

Sources: [Wikipedia: RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Chapter VIII · Confidently wrong

### The context window
*Chapter VIII · Memory*

A model has no memory of its own between conversations. It can only see the text in front of it.

Everything the model can use while answering has to fit in its context window: your messages, its own replies, and any documents you paste in. GPT-3 could see about two thousand tokens at once, a few pages of text. By 2024, some models accepted one hundred and twenty-eight thousand tokens, and Gemini 1.5 Pro up to a million, several long novels. Anything outside the window is simply not there for the model. What it “knows” beyond that is stored, blurrily, in its weights.

Chapter VIII · Step 10

#### What the model can see

The **context window** is the maximum number of tokens the model can attend to at once. A chat app usually re-sends the whole conversation each turn, so long chats eventually spill out of the window. Some chat apps add “memory” by saving notes and placing them back into the context.

Try it: window size

- **GPT-2 (2019)**: ~1,000 tokens (“1k”)

- **GPT-3 (2020)**: 2,048 tokens

- **GPT-4 (2023)**: 8K · 32K default · extended version

- **Gemini 1.5 Pro (2024)**: 1,000,000 tokens in the production API

Sources: [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model); [Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3); [Wikipedia: GPT-4](https://en.wikipedia.org/wiki/GPT-4); [Wikipedia: Llama](https://en.wikipedia.org/wiki/Llama_(language_model)); [Wikipedia: Gemini](https://en.wikipedia.org/wiki/Gemini_(language_model))

### Confidently wrong
*Chapter VIII · Hallucination*

The model always produces the next word. It has no built-in alarm that rings when it does not know.

Ask about something well known, and almost all the probability piles onto the right answer. Ask about something obscure, and the probability spreads thin across many plausible guesses. But the sampler still has to pick one. The result reads just as smoothly as a true answer. This is called a hallucination. In 2023 a lawyer in New York filed a brief citing six court cases that ChatGPT had invented. The judge fined the lawyers five thousand dollars.

Chapter VIII · Step 11 · 2023

#### Why models can be confidently wrong

A **hallucination** is a fluent, plausible-sounding answer that is false. It follows from how the model works: it is trained to produce likely text, not to check facts, and it must output something at every step. Researchers at OpenAI argued in 2025 that common training and test scoring also reward guessing over saying “I don’t know”.

Try it: well known vs obscure*hand-authored probabilities*
Look at the top bar. A spread-out distribution is a warning sign, but a chat interface normally shows only the chosen words, not the probabilities behind them.

- **Mata v. Avianca (New York, 2023)**: Six fabricated case citations; two lawyers fined $5,000 in June 2023 a well-known early example

What helps: giving the model reliable documents in its context, letting it search or calculate with tools (next chapter), training it to say when it is unsure, and checking important answers yourself.

Sources: [Wikipedia: Hallucination (AI)](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Chapter IX · Tools and thinking

### Thinking before answering
*Chapter IX · Tools and thinking*

Models that call calculators and search engines, and models that write out their reasoning before they reply.

A language model cannot do exact arithmetic reliably or know today’s news. But it can write text that asks a tool to do it. The app runs the tool, a calculator, a search, a piece of code, and pastes the result back into the context. The model carries on from there. Systems that chain many such steps towards a goal are called agents. In September 2024 OpenAI introduced o1, a reasoning model. Before answering, it generates a long chain of intermediate steps, and checks and revises them. Such models are trained with reinforcement learning on problems whose answers can be checked. Thinking longer often helps, but it costs much more computing.

Chapter IX · Step 12 · 2024 →

#### Tools, agents and reasoning models

**Tool use:** the model is trained to emit a structured request (“search for…”, “calculate…”). Software outside the model carries it out and returns the result as more context. **Agents** loop this: plan, act, observe, repeat. **Reasoning models** spend extra computation at answer time, writing a chain of thought before the final reply.

Try it: call a tool

- **o1-preview**: Sep 2024 OpenAI’s first reasoning model

- **DeepSeek-R1**: Jan 2025 reasoning model trained with reinforcement learning

- **The trade-off**: One study found reasoning models 10 to 74 times more expensive to run on a maths benchmark and they are slower to answer

The “reasoning” is itself generated text, produced the same way as any other answer. OpenAI chose not to show o1’s raw reasoning to users, returning a model-written summary instead.

Sources: [Wikipedia: Reasoning model](https://en.wikipedia.org/wiki/Reasoning_model); [Wikipedia: Intelligent agent](https://en.wikipedia.org/wiki/Intelligent_agent); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Chapter X · Open questions

### Costs, rights and risks
*Chapter X · Open questions*

The technology raises questions that engineering alone cannot answer.

Energy. Data centres used about one and a half per cent of the world’s electricity in 2024. The International Energy Agency projects that their use will more than double by 2030, driven largely by AI. Data and copyright. Models learn from text written by people. Authors and publishers have sued AI companies; the courts have not given one consistent answer yet. Safety and alignment. How do we make sure powerful models do what people intend? Some researchers see serious risks; others think those fears are overstated. The debate is open.

Chapter X · Step 13

#### Limits and open questions

These questions are debated by scientists, companies, governments and the public. Here are the main points, without taking sides.

#### Energy

The International Energy Agency estimated that data centres used about 415 TWh of electricity in 2024 (about 1.5% of the world total) and projected about 945 TWh by 2030, roughly Japan’s current consumption. Per-answer energy varies widely with the model and task; companies and researchers are working on more efficient chips and models.

#### Data and copyright

In December 2023 *The New York Times* sued Microsoft and OpenAI over the use of its articles in training. In June 2025 a US federal court ruled in *Bartz v. Anthropic* that training the company’s models on the authors’ books was fair use, but that using millions of pirated library copies was not; in September 2025 Anthropic agreed to pay $1.5 billion to settle the case. Other cases are still running, and the law differs between countries.

#### Safety and alignment

**Alignment** means steering AI systems towards the goals and values people intend. In 2023 many researchers and company leaders signed a one-sentence statement that “mitigating the risk of extinction from AI should be a global priority”; other prominent researchers argue such fears are exaggerated or premature. Nearer-term concerns include misinformation, bias, privacy and misuse.

Sources: [Scientific American on the IEA “Energy and AI” report (Apr 2025)](https://scientificamerican.com/article/ai-will-drive-doubling-of-data-center-energy-demand-by-2030); [Bloomberg Law, 27 Dec 2023](https://news.bloomberglaw.com/ip-law/new-york-times-sues-microsoft-openai-for-copyright-infringement); [Wikipedia: Bartz v. Anthropic](https://en.wikipedia.org/wiki/Bartz_v._Anthropic); [Wikipedia: AI alignment](https://en.wikipedia.org/wiki/AI_alignment); [Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)

## Epilogue

### Autocomplete, scaled up
*Epilogue*

A simple goal, predict the next token, plus vast data and computing power, produced something nobody fully predicted.

Text into tokens. Tokens into vectors. Attention between them, layer after layer. A probability for every possible next token. Billions of dials, tuned by gradient descent on trillions of words, then shaped by human feedback. The result can explain, translate, write code and reason through problems. It can also be confidently wrong. Knowing how it works is the best way to use it well: as a powerful tool, whose answers are worth checking.

Epilogue · 2026 →

#### How a chatbot thinks, in one page

- 1 Text is cut into **tokens** and each becomes a **vector**.

- 2 **Attention** layers mix information between tokens, dozens of times over.

- 3 The model outputs a **probability** for every possible next token; one is sampled.

- 4 All the weights were learned by **gradient descent** on next-token prediction.

- 5 **Instruction tuning** and **human feedback** turn it into an assistant.

- 6 **Tools** and **reasoning** extend what it can do; **hallucinations** remain a risk.

- 1986 Backpropagation popularised

- 2013 word2vec: meaning as vectors

- 2017 The Transformer

- 2020 GPT-3, 175 billion parameters; scaling laws

- 2022 InstructGPT, then ChatGPT

- 2024 Million-token context; first reasoning models

- 2026 GPT-6 rolls out in ChatGPT (October)

The 3D scenes are explanatory drawings: token ids, embedding positions, attention weights and probabilities in the hallucination step are illustrative, except where a number is quoted from a source. The next-word demo is a real (tiny) trigram model trained in your browser on public-domain text. All full-screen stills are AI-generated to help the imagination and are not photographs.

#### Sources

[Wikipedia: Large language model](https://en.wikipedia.org/wiki/Large_language_model)
[Wikipedia: ChatGPT](https://en.wikipedia.org/wiki/ChatGPT)
[TechCrunch, 7 Oct 2026](https://techcrunch.com/2026/10/07/chatgpt-is-getting-a-lot-more-visual-with-the-launch-of-a-new-interface/)
[OpenAI Help Center: What are tokens](https://help.openai.com/en/articles/4936856-what-are-tokens)
[Wikipedia: Byte-pair encoding](https://en.wikipedia.org/wiki/Byte-pair_encoding)
[Mikolov, Yih & Zweig, NAACL-HLT 2013](https://aclanthology.org/N13-1090.pdf)
[Wikipedia: Word2vec](https://en.wikipedia.org/wiki/Word2vec)
[Wikipedia: Word embedding](https://en.wikipedia.org/wiki/Word_embedding)
[Wikipedia: Softmax function](https://en.wikipedia.org/wiki/Softmax_function)
[Wikipedia: Backpropagation](https://en.wikipedia.org/wiki/Backpropagation)
[Wikipedia: Gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)
[Wikipedia: Attention Is All You Need](https://en.wikipedia.org/wiki/Attention_Is_All_You_Need)
[Wikipedia: Transformer (deep learning)](https://en.wikipedia.org/wiki/Transformer_(deep_learning))
[Wikipedia: GPT-3](https://en.wikipedia.org/wiki/GPT-3)
[Wikipedia: GPT-4](https://en.wikipedia.org/wiki/GPT-4)
[Kaplan et al., Scaling Laws for Neural Language Models (2020)](https://arxiv.org/abs/2001.08361)
[Wikipedia: Neural scaling law](https://en.wikipedia.org/wiki/Neural_scaling_law)
[Wikipedia: Chinchilla](https://en.wikipedia.org/wiki/Chinchilla_(language_model))
[Wikipedia: Llama](https://en.wikipedia.org/wiki/Llama_(language_model))
[Wikipedia: Hopper / H100](https://en.wikipedia.org/wiki/Nvidia_H100)
[Wikipedia: Mixture of experts](https://en.wikipedia.org/wiki/Mixture_of_experts)
[Wikipedia: RLHF](https://en.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback)
[Wikipedia: Gemini](https://en.wikipedia.org/wiki/Gemini_(language_model))
[Wikipedia: Hallucination (AI)](https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence))
[Wikipedia: Reasoning model](https://en.wikipedia.org/wiki/Reasoning_model)
[Wikipedia: Intelligent agent](https://en.wikipedia.org/wiki/Intelligent_agent)
[Scientific American on the IEA “Energy and AI” report (Apr 2025)](https://scientificamerican.com/article/ai-will-drive-doubling-of-data-center-energy-demand-by-2030)
[Bloomberg Law, 27 Dec 2023](https://news.bloomberglaw.com/ip-law/new-york-times-sues-microsoft-openai-for-copyright-infringement)
[Wikipedia: Bartz v. Anthropic](https://en.wikipedia.org/wiki/Bartz_v._Anthropic)
[Wikipedia: AI alignment](https://en.wikipedia.org/wiki/AI_alignment)
[Project Gutenberg eBook #11](https://www.gutenberg.org/ebooks/11)