How does an AI pick its next word?

The number came up at a dinner

Reported November 3, 2026

It was an OpenAI-hosted dinner in New York, and the speaker was Adam Fry, ChatGPT's consumer product lead. ChatGPT, he said, had gone from 1 billion weekly users to 1.2 billion in 60 days. Source OpenAI had already shown the 1.2 billion at its DevDay event on September 29. Source It is OpenAI's own count, and the reports cite no outside check.

Two hundred million more people in two months, all typing into the same box.

Here is a smaller question, the kind that doesn't usually come up at dinner.

There is no answer waiting anywhere

A chatbot writes one small piece of text and adds it to what is already there. Then it looks at the whole thing and picks the next piece. Then it does that again, until it stops. Researchers call this autoregressive, which means each output becomes part of the next input. Source Textbook

The survey you just saw, by the Searchlight Institute and Tavern Research, asked 2,301 US adults in August 2025. Survey Analysis The biggest group, 45%, were picturing a very large filing cabinet.

A chatbot can also search the web and read what it finds. It still writes the reply this way, one piece at a time.

The pieces are called tokens

A token is a chunk of text from a fixed list. Some tokens are whole words. Others are common bits, like -er. Source

One OpenAI tokenizer splits tiktoken is great! into six tokens: t ikt oken  is  great ! Notice the spaces. Here, each space is part of the token after it. Source

The splits depend on the list. antidisestablishmentarianism is 5 tokens in an older OpenAI list and 6 in newer ones. So think of a token as roughly a word piece.

Every token gets a chance

At each step, the model gives a score to every token on its list. GPT-2, an older OpenAI model, has a list of 50,257. Source A piece of math called softmax turns those scores into probabilities that add up to 100%. Source

Even a silly token gets some tiny chance. In a model like this, the number is never exactly zero.

Then it picks one, and not always the top one

The simplest rule is to always take the top token. That is called greedy, and it gives generic, repetitive text. So the usual choice is a draw, weighted by the odds. That is why the same question can get two different answers. Source

Temperature changes how even the odds are before the draw. It never changes the order, only the gaps.

We found no temperature setting in the ChatGPT or Claude apps. Developers can't change it on newer Claude models, and on the newest OpenAI models only in limited cases. Anthropic says its newer models run at a temperature of 1. You can try the math yourself in open models and demos. Anthropic OpenAI

Train your own predictor

This one reads a text, counts which word follows which, and turns the counts into odds. It is called a bigram model, and it runs on your device. Claude Shannon printed text like this in 1948, and it still reads the way he showed: fine up close, nonsense from a distance. Source

The sample is the first two chapters of Alice's Adventures in Wonderland. Press Train it. Pick a few words. Add 20 and read them out loud. Then drag the temperature all the way down, and all the way up.

What your predictor shares with a real model

More than you might expect.

• Both give odds for each item on a fixed list.
• Both draw one, add it, and repeat.
• Both start from text, with no grammar rules written by hand.
• Both get the same grade: how surprised they were by the true next item.
• Both are shaped by what they read.

For a table like yours, counting and dividing gets the best possible score on that grade for the text it trained on, in one step. Source

Same job. Same grade sheet. It does not have the same insides.

Why a bigger table can't get there

Your table looks back one word. A word-pair table over GPT-2's list would have about 2.53 billion cells. That is already more numbers than all of GPT-2, which has 1.5 billion. A table for the 1,024 tokens GPT-2 can look back on would need about 10⁴⁸¹⁴ rows. No computer gets there. Source

So a real model stores weights instead. Weights are numbers the model adjusts during training, and they are shared across every situation. None of them reads as "the odds of mat." The odds get worked out fresh at each step. Source

That also fixes a problem you may have hit. Your table has nothing to say about a pair it never saw. A real model makes an estimate from similar contexts. Source

How does it learn?

Training shows the model real text and, at every position, asks for odds on what comes next. Then it checks one number: how much probability went to the true next token. That score is called loss, and it measures surprise. Source

The weights get nudged to lower the loss, and the process repeats. GPT-3 models trained for 300 billion tokens. Source Meta's Llama 3.1 flagship trained on 15.6 trillion. Source We could not find an exact number for the newest top models from OpenAI, Anthropic or Google. OpenAI's public summary for GPT-6 Astra says only: more than 10 trillion tokens of text. Source Anthropic's summary for Claude Opus 5.5 says the same. Source

Why does it answer you instead of just continuing?

Next-word training alone gives a base model, and base models are poor at following instructions. In a hand-picked example from a 2022 OpenAI paper, a request in French for a short story about a time-traveling frog went to GPT-3. It wrote more writing prompts. The tuned version, InstructGPT, wrote the story.

The tuning took people. Labelers wrote good answers, and the model was tuned on them. They ranked outputs, a second model learned their preferences, and the first was then trained to score well with it. Source

That did not make it truthful. On tasks where the answer should come from a given text, that 2022 version still made up information 21% of the time, against 41% for GPT-3. In a 2026 Common Sense Media survey, 42% of US kids aged 9 to 17 said AI can tell true from false. Source

Is "it just predicts the next word" the whole story?

It's accurate about the output. Whether it covers what happens inside is a live argument.

In 2025, Anthropic, the company that makes Claude, studied one of its own models writing a rhyming couplet. The first line ended in "grab it," and the second ended in "rabbit." The authors expected the model to improvise at the end of the line. They found signals for "rabbit" and "habit" already active before the line began. Changing those signals changed the line. Source

They found these signals in about half the poems they checked. Their tool gave a satisfying look inside on about a quarter of the prompts they tried. They show only successes, and even those capture a small part of the model's work. Source A 2026 paper by European and Google DeepMind researchers calls this kind of planning universal. Source Two Stanford researchers tested more than ten open models, and only Google's Gemma-3-27B clearly used the signal at the line break. Source Part of the disagreement is about what counts as planning. None of this shows intention or awareness.

Build it on a whole book

• You will have A word-pair table from a real book, text it wrote, and a side-by-side with GPT-2.
• Time About 60 to 90 minutes.
• You need A computer, JupyterLite (free, no account, runs in your browser), and an adult for steps 1 and 6.

1. Ask a parent or teacher to download one plain-text book from Project Gutenberg. Its terms say minors should be accompanied by an adult. Pick a book in English, since this code only reads a to z.
2. Delete Gutenberg's notes at the top and bottom of the file, or they end up in your counts. Name it book.txt.
3. Open JupyterLite, start a Python notebook, upload the file, and run this:

import random, re
from collections import Counter, defaultdict

text = open('book.txt', encoding='utf-8').read().lower()
words = re.findall(r"[a-z']+", text)

table = defaultdict(Counter)
for a, b in zip(words, words[1:]):
    table[a][b] += 1

def top(word, n=3):
    c = table[word]
    total = sum(c.values())
    return [(w, round(k / total * 100, 1)) for w, k in c.most_common(n)]

def write(start, length=8, seed=1):
    random.seed(seed)
    out = [start]
    for _ in range(length):
        c = table[out[-1]]
        if not c:
            break
        out.append(random.choices(list(c), weights=list(c.values()))[0])
    return ' '.join(out)

print(top('the'))
print(write('the', 20))

4. Try other words. Find one with a surprising favorite. Change seed and watch the text change.
5. Open Transformer Explainer to see the real odds from a small version of GPT-2. It downloads about 600 MB, so use Wi-Fi and ask first.
6. Have an adult type your same starting words into a big chatbot, on their own account. Many services are 13+, often with a parent's permission, and some, including Claude, are 18+.
7. Write down one place each of the three failed, and why. Save your notebook before you close the tab, because browser storage can get wiped.

Back to the box

Tonight a great many people will type a question into that box. It will answer in whole sentences, one piece at a time, and it will sound sure of itself.

You can keep using it. You just know what it is doing now.

Back to the Lab

Last checked November 4, 2026. This field moves fast, so some model details may have changed. Torah Tech researched and wrote this lesson with the help of Claude, an AI made by Anthropic, whose research appears above. Any mistakes are ours.