AI Adoption

Today’s lecture

1. Introduction
2. Pretraining data
3. Tokenization
4. The prediction task
5. The Transformer
6. The base model
7. Supervised fine-tuning
8. Consequences of training
9. Reinforcement learning

The Common Crawl archive

The crawl of September 2026, CC-MAIN-2026-39, contains 2.17 billion pages, 361.4 TiB of uncompressed content. It is distributed in three file formats:

The pretraining data of Llama 3 (2024)

Llama 3 405B was trained on \(15.6\cdot10^{12}\) tokens of text with knowledge up to the end of 2023.

The pretraining data of Olmo 3 (2025)

Olmo 3 32B was pretrained on \(5.5\cdot10^{12}\) tokens.

The pretraining data of MAI-Thinking-1 (2026)

Microsoft, 35B active of 1T parameters: \(30\cdot10^{12}\) tokens.

Building a pretraining dataset is highly nontrivial!

Vocabulary size and sequence length

Setting. A text is a byte string \(s\in\Sigma^{*}\), \(\Sigma=\{0,\dots,255\}\).
A vocabulary is a finite set \(\mathcal V\subset\Sigma^{+}\) of tokens.

Definition. A tokenizer is a function \(\tau:\Sigma^{*}\to\mathcal V^{*}\) with \(\tau(s)=(x_1,\dots,x_T)\) and \(x_1x_2\cdots x_T=s\).

Remark. Tokens make the matrices of the Transformer smaller.

Byte-pair encoding: training

Tokenizers of real models

Language models

Setting. \(\mathcal V\) is a vocabulary; its elements are called tokens.
A text is a sequence \(x=(x_1,\dots,x_T)\in\mathcal V^T\).
We write \(\Delta(\mathcal V)\) for the set of probability distributions on \(\mathcal V\).

Definition. A language model with parameters \(\theta\in\mathbb R^N\) is a map that assigns to every prefix a probability distribution of the next token:

Remark. By the chain rule, \(p_\theta\) defines a probability distribution on texts of each length \(T\):

The cross-entropy loss

\(x=(x_1,\dots,x_M)\in\mathcal V^M\)

the training corpus of \(M\) tokens: all documents, concatenated

\(n\in\mathbb N\)

the context length; we assume that \(n\) divides \(M\) and set \(J=M/n\)

\(b^{(j)}\in\mathcal V^n\)

block \(j\) of the corpus, \(b^{(j)}=(x_{(j-1)n+1},\dots,x_{jn})\), for \(j=1,\dots,J\)

Definition. The cross-entropy loss \(\mathcal L:\mathbb R^N\to[0,\infty)\), in nats per token, is

Definition. The perplexity is \(e^{\mathcal L(\theta)}\), i.e. we can think that the model is as uncertain as a uniform choice among \(e^{\mathcal L}\) tokens (for example, \(\ln 10 \approx 2.3\) nats per token, means that perplexity is 10).

What the network computes

Definition. \(\operatorname{softmax}:\mathbb R^{|\mathcal V|}\to\Delta(\mathcal V)\), \(\operatorname{softmax}(z)_v=e^{z_v}/\sum_{u\in\mathcal V}e^{z_u}\);.

Why a network is mostly matrices

Theorem (universal approximation; Cybenko 1989, Hornik 1991, Leshno et al. 1993).
Let \(\sigma:\mathbb R\to\mathbb R\) be continuous and not a polynomial.
For every compact \(K\subset\mathbb R^n\), continuous \(f:K\to\mathbb R^m\) and \(\varepsilon>0\) there are \(k\in\mathbb N\), \(W_1\in\mathbb R^{k\times n}\), \(b\in\mathbb R^k\), \(W_2\in\mathbb R^{m\times k}\) with

Examples of \(\sigma\): ReLU \(\max(0,t)\); sigmoid \(1/(1+e^{-t})\); SiLU \(t/(1+e^{-t})\) (in the MLP of Llama).

MLP

Note: \(u_k\) is a key, a pattern, and \(w_k\) its value.

A Transformer

Generating text: autoregressive sampling

The whole algorithm in one line: draw the next token from the model’s probabilities, write it at the end of the window, and drop the leftmost token if the window is full. Repeat until the model draws the end-of-text token or the length limit is reached.

for _ in range(max_new_tokens):
    window = idx[:, -block_size:]                       # the last block_size tokens
    logits, _ = model(window)                           # one pass of the network
    probs = F.softmax(logits[:, -1, :], dim=-1)         # probabilities of the next token
    idx_next = torch.multinomial(probs, num_samples=1)  # draw one token
    idx = torch.cat((idx, idx_next), dim=1)             # write it at the end

What a base model computes

The base model is the network right after pretraining. It was trained only to continue documents, so it simulates documents: given the beginning of one, it writes a plausible rest.

• It continues a prompt the way training documents that begin with this prompt continue.
• Each run samples another continuation: another plausible document.
• If it predicted every next token exactly as in its data, a sampled continuation would have exactly the probability of that continuation among the training documents (the chain rule of Section 4).

Remark. The objective does not mention questions: a question in the prompt is continued as documents that begin with it continue, by an answer, more questions, or comments.

Temperature

Before the draw, the scores of the tokens (the logits) are divided by the temperature and turned into probabilities again.

• Low temperature: the most likely token takes almost everything; temperature 0 is greedy decoding.
• Temperature 1: the model’s own probabilities.
• High temperature: the probabilities become more even, and unlikely tokens are drawn more often.

Example. Tokens after “… exercise to the”.

Truncated sampling: top-k, top-p, min-p

Sampling occasionally draws a very unlikely token, and one bad token can derail the rest of the text. Truncation keeps only the plausible tokens, rescales their probabilities so that they sum to 1, and draws among them.

Usually the temperature is applied first, then the truncation.

Memorisation of training text

A passage of training text is extractable if, given the tokens that come before it in the training data, the model, always taking its most likely token, writes the passage out word for word (Carlini et al. 2023).

On GPT-Neo models (125M to 6B parameters, 825 GB of text), For the 6B model, 33% of the tested passages are extractable from 50 tokens of context and 65% from 450.

How much the weights can hold (Llama 3 8B). 8 billion parameters of 16 bits each, trained on over 15 trillion tokens:

16 bits × 8 billion / 15 trillion tokens ≈ 0.009 bits per training token.

The knowledge cutoff

Definition. The knowledge cutoff of a model is the date of the latest text in its pretraining data.

In-context learning

Example (GPT-3, 175B parameters, not trained further). The prompt contains solved pairs (sentence, translation); the score is BLEU, from 0 to 100, on the WMT’14 test set.

The cost of training GPT-2

Training a base model of GPT-2’s quality (1.5B parameters, 2019) from scratch:

The system prompt

The system prompt is the first message of a conversation, with role system, written by the developer of the application. The chat template puts it at the start of the token sequence, before the user’s messages and the replies.

Remark. Nothing in the network marks the system prompt as special: i.e., the instruction hierarchy, is learned from fine-tuning data with conflicting instructions.

The supervised fine-tuning objective

Supervised fine-tuning (SFT) continues training the base model with the same next-token loss as pretraining, on conversations instead of web documents.

Where the conversations come from

Hallucinations

Definition. A hallucination is a fluent and specific reply that is false, such as a wrong date, name, number or reference, stated in the same form as a correct reply.

Demonstration. In the terminal: 8, the same question three times to OLMo 2 7B, in the dialogue prompt of Question 5.

Remedy. Add to the fine-tuning data conversations in which the model does not know the answer and says so, for example:

User: When is the birthday of Adam Tauman Kalai?
Assistant: I don’t know his birthday.

Tool use

A tool call is text that the model writes and the application executes (Llama 3.1 format):

1. the model writes the special token <|python_tag|>, then a call, then <|eom_id|>, “end of message”;
2. generation pauses; the application runs the call (a search, a calculator, code);
3. the result is appended to the context as a message of the role ipython, followed by a new assistant header;
4. the model continues, now reading the result, until it ends its reply with <|eot_id|>.

Example of a call: <|python_tag|>brave_search.call(query="...")<|eom_id|>

Tokenization and spelling

A token enters the network as one learned vector, its row of the embedding table; its letters are not part of the input.

Example (o200k_base, GPT-4o; a box is one token, spaces included):

Example. Asked “How many Ds are in DEEPSEEK?”, DeepSeek-V3 answered 2 or 3 in ten trials (the answer is 1); DeepSeek-R1 first spelled the word and answered 1.

Tokenization of numbers

Example (o200k_base; a box is one token, spaces included; ␣ is the token consisting of one space). “9.11 > 9.9” and “12345 * 6789 =”:

Sycophancy and prompt injection

Three kinds of training data

DeepSeek-R1-Zero: reinforcement learning without supervised data

Remark. Reflective words (“wait”, “verify”, “check”, “mistake”, …) become 5 to 7 times more frequent.

Training a model to reason

A reasoning model writes a chain of thought before its answer. Example (constructed):

User: What is the sum of the digits of 2^10?
Assistant: <think>2^10 = 1024. Digits 1, 0, 2, 4, sum 7. Wait, check 2^10:
2^5 = 32, and 32 · 32 = 1024. Yes.</think> The sum is 7.

Remark. Fine-tuning teaches the form of a reasoning text; RL makes the reasoning that reaches correct answers more likely.

Controlling the length of reasoning

A reasoning model first writes its thinking between <think> and </think>, then the answer.

Human judgements with and without a reward model

A prompt such as “Write a limerick about a topologist” has no answer that a program can check. A reward model, a network trained on human comparisons, gives each response a score.

Example (illustrative numbers). Human judgements for one RL run, without and with a reward model: