Foundation model training: a field guide

0. Orientation: how a foundation model gets made

A model like Llama 3 or DeepSeek-R1 comes out of a pipe­line whose stages cost very dif­fer­ent amounts. Start by sep­a­rat­ing the stages, the costs each report counts, and the work its head­line total leaves out.

What does it take to make a model like Llama 3 or DeepSeek-R1, and where does the effort go? A foun­da­tion model is a large model pre­trained on broad data and then adapted to many tasks. It is the output of a pipe­line. You col­lect and filter text, pre­train on it, mid­train on a curated mix, post-train for behav­ior, eval­u­ate at every step and then serve. This guide fol­lows a few pub­lished runs through every stage.

DeepSeek's own account­ing shows the shape of the effort. DeepSeek-V3 took 2,788K H800 GPU-hours. The H800 is a Hopper GPU like the H100 with NVLink cut from 900 to 400 GB/s [1], and a GPU-hour is one accel­er­a­tor busy for one hour. Of those, 2,664K went to pre­train­ing, 119K to extend­ing the con­text window and 5K to post-train­ing (reported) [2]. DeepSeek-R1 then built on V3 and added 147K GPU-hours of rea­son­ing train­ing (reported) [3]. Pre­train­ing alone was 2,664 / 2,788 = 95.6% of V3's budget (com­puted).

The small later budget is not small in effect. During the rein­force­ment learn­ing (RL) that pro­duced R1-Zero, a model trained with RL alone from the V3 base, the model's aver­age pass@1 on the AIME 2024 math com­pe­ti­tion, the share of prob­lems solved with a single sample, rose from 15.6% to 77.9% (reported) [3].¹ That RL run cost 101K GPU-hours, or 101 / 2,788 = 3.6% of V3's total (com­puted).

The chap­ter has four parts. Sec­tion 0.1 draws the pipe­line and links each stage to its chap­ter. Sec­tion 0.2 prices the stages in com­pute, time and people. Sec­tion 0.3 intro­duces the run­ning exam­ples and the nota­tion, and Sec­tion 0.4 sug­gests where to start.

0.1 The pipeline in one picture

What are the stages, in what order, and which chap­ter covers each? Figure 1 draws them from left to right, from web text to a served model. Every box links to its chap­ter, so the figure dou­bles as a visual table of con­tents. Switch between the Llama 3 and DeepSeek pre­sets to read one or two reported num­bers per stage, and follow the dashed arrows: they are the loops that make the pipe­line less linear than it looks.

The base layer: com­pute, memory and hard­ware

Every stage is paid for in FLOPs (float­ing-point oper­a­tions), bytes and band­width, and Chap­ter 1 and Chap­ter 2 teach that arith­metic and the machines behind it. The ref­er­ence accel­er­a­tor in this guide, the H100 SXM, does 989.5 TFLOP/s of dense math in BF16, a 16-bit float­ing-point format (Sec­tion 2.3), and reads its high-band­width memory (HBM) at 3.35 TB/s [4]. Its datasheet prints 1,979 TFLOP/s, a figure that assumes struc­tured spar­sity, so the dense rate is half of it (com­puted).

Data and the tok­enizer

Chap­ter 3 turns the raw web into train­ing tokens. FineWeb, an open dataset, fil­ters 96 Common Crawl snap­shots down to 15T tokens [5]. That is about the size of one fron­tier run, though the two counts use dif­fer­ent tok­eniz­ers: Llama 3 405B trained on 15.6T text tokens [6]. A tok­enizer maps text to inte­gers (Sec­tion 3.7). Llama 3's aver­ages 3.94 char­ac­ters per token on Eng­lish, up from 3.17 for Llama 2 [6].

Archi­tec­ture and pre­train­ing

Chap­ter 4 covers the net­work. Dense decoders such as Llama 3 use every param­e­ter on every token. Mix­tures of experts such as DeepSeek-V3 store 671B param­e­ters and acti­vate only 37B per token [2]. Pre­train­ing teaches the net­work to pre­dict the next token (Sec­tion 4.1), with the opti­mizer and sched­ules of Chap­ter 5, at a size chosen in advance with scal­ing laws (Chap­ter 6). Llama 3 405B used $3.8\times {10}^{25}$ FLOPs on up to 16,384 H100s at 38⁠–⁠43% model FLOPs uti­liza­tion (MFU), the share of the dense BF16 peak spent on the model's own math [6]. Meta's fitted scal­ing law had sug­gested 402B param­e­ters on 16.55T tokens for that budget [6], close to the 405B on 15.6T it trained.

Sys­tems: spread­ing the run and keep­ing it alive

In BF16 the 405B model's weights alone take $2\times 405.85\times {10}^9\approx 812$ GB (com­puted), ten times an H100's 80 GB [4]. So Chap­ter 11 splits each step across devices. The 405B run com­bined 8-way tensor, 16-way pipe­line and 128-way data par­al­lel­ism on 16,384 GPUs [6]. Chap­ter 12 keeps such a run alive. In a 54-day snap­shot of pre­train­ing it had 466 job inter­rup­tions and still achieved more than 90% effec­tive train­ing time [6].

Mid­train­ing

Mid­train­ing (Chap­ter 7) sits between bulk pre­train­ing and post-train­ing. It lowers (anneals) the learn­ing rate toward zero while train­ing on curated data, and extends the con­text window. Llama 3 405B's long-con­text stage used about 800B tokens [6], and OLMo 2 7B's score on the GSM8K math bench­mark rose from 24.1 to 67.5 through mid­train­ing [7].

Post-train­ing and rein­force­ment learn­ing

Post-train­ing (Chap­ter 8) teaches format and behav­ior. Its tools are super­vised fine-tuning (SFT), reward models, RL from human feed­back (RLHF) and direct pref­er­ence opti­miza­tion (DPO). Its lever­age showed early: label­ers pre­ferred the out­puts of a 1.3B InstructGPT model to those of the 175B GPT-3 [8]. Llama 3 used SFT, rejec­tion sam­pling (keep­ing the best of sev­eral sam­pled answers) and DPO, repeated over six rounds [6]. Chap­ter 9 covers RL with rewards a pro­gram can check, the route to rea­son­ing models that write a long chain of thought before they answer. R1-Zero's AIME jump took 101K GPU-hours [3].

Eval­u­a­tion, serv­ing and sand­boxes

Eval­u­a­tion (Chap­ter 10) is not a final exam. It runs at every stage, and every score needs an error bar (Sec­tion 10.4). Llama 3 reports its 8B base model's HumanEval coding score as 37.2 ± 7.4, a 95% inter­val [6]. Serv­ing (Chap­ter 13) is paid every day, and Sec­tion 0.2 com­pares a day of it with train­ing. Sand­boxes (Chap­ter 14) are iso­lated envi­ron­ments where model-writ­ten code runs. DeepSWE, an open coding agent trained with RL, spawned 512 con­tain­ers per RL iter­a­tion [9], and DSec, DeepSeek's sand­box plat­form, sup­ports over 380,000 con­cur­rent sand­boxes [10]. Chap­ter 15 then plans a whole run.

The loops

The dashed arrows matter as much as the solid ones. Eval­u­a­tion steers every stage: small-scale abla­tions choose the data (Sec­tion 3.9), bench­mark scores pick anneals and check­points (Sec­tion 7.3), and each post-train­ing round is judged before the next one starts (Sec­tion 8.8). RL rewards are them­selves auto­matic eval­u­a­tions (Sec­tion 9.1). RL also ties post-train­ing to infer­ence, because gen­er­at­ing roll­outs is decod­ing (Sec­tion 9.8 and Sec­tion 13.2), and to sand­boxes, because the rewards come from run­ning code (Sec­tion 14.6 and Sec­tion 14.7). Finally, usage and fail­ures in serv­ing shape the data and the goals of the next model.

With the stages named, Sec­tion 0.2 asks what each one costs.

0.2 What each stage costs: compute, time and people

How much com­pute and wall-clock time does each stage take, and how many people, by the labs' own num­bers? We count in three cur­ren­cies: GPU-hours or FLOPs, cal­en­dar time, and people. Figure 2 plots com­pute by stage on a log scale, so that the small stages stay vis­i­ble next to pre­train­ing. Its pre­sets switch between DeepSeek-V3 and R1, Llama 3.1, InstructGPT, SmolLM3 and OLMo 2. In the pre­sets that split com­pute by stage (DeepSeek, InstructGPT and OLMo 2), pre­train­ing sits 20 to 750 times above every later stage (com­puted). The SmolLM3 preset shows what such totals leave out.

Com­pute

DeepSeek gives the clean­est split by stage. At an assumed rental price of $2 per H800 GPU-hour, V3's 2,788K GPU-hours come to $5.576M (reported) [2], and R1's 147K come to $294K (reported) [3].

Llama 3.1 used 39.3M H100 GPU-hours in all: 1.46M for the 8B model, 7.0M for the 70B and 30.84M for the 405B (reported) [11]. The 405B took 30.84 / 39.3 = 78.5% (com­puted). At DeepSeek's $2 per GPU-hour, used here only as an illus­tra­tion, its hours alone would rent for 30.84M × $2 = $61.7M (com­puted). The model card gives one total per model and does not split pre­train­ing from post-train­ing, so it cannot show stage shares.

InstructGPT reports its costs in PF/s-days, one petaFLOP/s sus­tained for a day, or $8.64\times {10}^{19}$ FLOPs. Its 175B SFT model needed 4.9 PF/s-days and its PPO-ptx model (RLHF with PPO, with pre­train­ing gra­di­ents mixed in) 60, against 3,640 for pre­train­ing GPT-3 [8]. That is 4.9 / 3,640 = 0.13% and 60 / 3,640 = 1.65%, or 1.78% together (com­puted).

What the totals leave out

These totals cover the offi­cial train­ing runs. DeepSeek's figure excludes "the costs asso­ci­ated with prior research and abla­tion exper­i­ments on archi­tec­tures, algo­rithms, or data" [2]. R1 was pre­pared with exper­i­ments on a smaller 30B model on A100 GPUs, which its table does not count [3]. The Olmo 3 report makes the same point about post-train­ing and eval­u­a­tion: "These addi­tional costs are not cap­tured when report­ing pre­train­ing hours alone" [12]. Sec­tion 12.8 accounts for every GPU-hour, and Sec­tion 15.1 shows how to spend a research budget.

Serv­ing catches up

Train­ing is paid once, and serv­ing is paid every day. DeepSeek's V3 and R1 ser­vice aver­aged 226.75 nodes over 24 hours [14]. At eight GPUs per node that is 226.75 × 8 × 24 = 43,536 H800 GPU-hours a day (com­puted), and at $2 per GPU-hour it gives the $87,072 daily cost that DeepSeek reports [14]. A day of serv­ing is 43,536 / 2,788,000 = 1.56% of V3's train­ing, so 2,788,000 / 43,536 = 64 days of serv­ing use as many GPU-hours as the whole train­ing run (com­puted). The ser­vice read 608B input tokens that day, 56.3% of them from an on-disk cache, and wrote 168B [14]. Sec­tion 13.3 turns such num­bers into the price of a token.

Time

Time to train is com­pute divided by deliv­ered FLOP/s, $T=C/(pF\text{MFU})$, derived in Sec­tion 1.3. For Llama 3 405B at 41% MFU on 16,384 H100s [6], the com­puted time is

The model card's 30.84M GPU-hours spread over 16,384 GPUs give $30.84\times {10}^6/\mathrm{16,384}\approx \mathrm{1,882}$ hours, or 78 days (com­puted). The report gives nei­ther figure as the wall-clock time. Read 66 days as the time the FLOPs need at the reported pace, and 78 days as every­thing the card charges to the model, in days of the full 16,384-GPU clus­ter. Part of the run used 8,192 GPUs [6], so nei­ther is the cal­en­dar time.

DeepSeek-V3's pre­train­ing fin­ished in "less than two months" [2]. Each tril­lion tokens took 3.7 days on a clus­ter of 2,048 H800 GPUs [2], and 3.7 × 14.8 ≈ 55 days for the 14.8T-token run (com­puted). Olmo 3 took about 56 days from the start of train­ing to the eval­u­a­tion of its 32B Think check­point on 1,024 H100s, a span that includes post-train­ing [12]. DeepSWE trained for six days on 64 H100s [9]. At the small end, llm.c repro­duces GPT-2 small in 90 min­utes for about $20 [15].

People

Llama 3's report says the model is "the result of the work of a large number of people at Meta" [6]. Its lists hold 236 core con­trib­u­tors, who worked on the project for at least two thirds of its run­time, and 323 con­trib­u­tors, who worked at least one fifth of it (com­puted: counted from the author lists). DeepSeek-V3 lists 150 people in research and engi­neer­ing, 31 in data anno­ta­tion and 18 in busi­ness and com­pli­ance (com­puted: counted from the author list) [2]. These are author lists, not team sizes or full-time equiv­a­lents. Much human work sits out­side them. InstructGPT hired a team of about 40 con­trac­tors to label data [8], and 93 soft­ware devel­op­ers screened the sam­ples of SWE-bench Ver­i­fied [16]. Auto­ma­tion stretches people: Llama 3 needed sig­nif­i­cant manual inter­ven­tion only three times in its 54-day snap­shot [6].

In every report that splits com­pute by stage, pre­train­ing comes first by a factor of 20 or more (com­puted: 2,664 / 119 = 22 for DeepSeek-V3, 3,640 / 60 = 61 for InstructGPT). Effort in people, data and research is spread far more evenly. That is why the chap­ters on data, post-train­ing, eval­u­a­tion and prac­tice (Chap­ter 3, Chap­ters 8 to 10 and Chap­ter 15) matter as much as Chap­ter 5. Sec­tion 9.4 tells the R1 story in full.

0.3 The running examples and the notation

Which models will you meet in every chap­ter, and what do the sym­bols mean? A small set of pub­lished models, the run­ning exam­ples, car­ries through the guide. When every chap­ter uses the same con­fig­u­ra­tions, its arith­metic can be checked against the others: the FLOPs of Chap­ter 1, the par­al­lel layout of Chap­ter 11 and the KV cache of Chap­ter 13 all describe the same Llama 3 70B. Param­e­ter counts are com­puted from the released con­fig­u­ra­tion files, so they match the check­points. Sec­tion 1.1 derives them, and Sec­tion 4.7 explains why a mix­ture of experts has two counts.

Each exam­ple has a job. Llama 3 8B, 70B and 405B are the dense work­horses for FLOPs, memory, par­al­lel­ism, fail­ures and infer­ence. DeepSeek-V3 is the mix­ture-of-experts exam­ple, trained in FP8 [2], and R1 the rea­son­ing model built on it [3]. GPT-2 small is the scale you can train and ablate on one node, and OLMo 2 is the run whose data and recipe are fully open [7].

One accel­er­a­tor serves as the hard­ware ref­er­ence through­out: the H100 SXM, with 989.5 TFLOP/s of dense BF16 com­pute (1,979 with spar­sity) and 80 GB of HBM at 3.35 TB/s [4]. The price of $2 per GPU-hour is DeepSeek's assump­tion [2] and serves only as an illus­tra­tion. Table 0.2 fixes the sym­bols. Each has one mean­ing on the whole page.

Two unit rules hold every­where. FLOPs is a count and FLOP/s a rate. GB is ${10}^9$ bytes and GiB is $2^{30}$ bytes, and one com­par­i­son never mixes them. In token counts, 8K means 8,192 and 128K means 131,072. Every number car­ries a label: (reported) for a source's own number, (com­puted) for arith­metic done here from cited inputs, and (illus­tra­tive) for an assumed value. Sec­tion 16.1 col­lects the for­mu­las and Sec­tion 16.2 defines the terms.

0.4 How to read this guide

Where should you start, and what can you skip on a first pass? The guide is writ­ten for engi­neers and sci­en­tists who know cal­cu­lus, linear alge­bra, prob­a­bil­ity and Python and want the arith­metic and prac­tice of train­ing and serv­ing models at scale. Each for­mula is derived once, in its home sec­tion, and col­lected in Sec­tion 16.1. Sec­tion 16.2 defines every term. Each number has a source and a label. Sec­tion 16.3 lists fifty sources worth read­ing in full.

The fig­ures are meant to be used. Hover over or focus any mark to read its value and source. Every figure has a table view with the num­bers it plots, and the cal­cu­la­tors take your own inputs, so you can put your model where the run­ning exam­ples sit.

A first pass covers about a fifth of the guide. After this chap­ter, read:

• Foun­da­tions, first if you have never writ­ten out a trans­former layer: Sec­tions 4.1⁠–⁠4.2 (the objec­tive and one Llama 3 8B block).
• Com­pute: Sec­tions 1.1⁠–⁠1.3, 1.7 and 2.1.
• Pre­train­ing: Sec­tions 3.2, 5.1⁠–⁠5.2, 6.1⁠–⁠6.2 and 7.1.
• Post-train­ing and eval­u­a­tion: Sec­tions 8.1⁠–⁠8.2, 8.5, 8.7, 9.1⁠–⁠9.4 and 10.4.
• Sys­tems and serv­ing: Sec­tions 11.1⁠–⁠11.2, 12.1, 13.1⁠–⁠13.2 and 15.4.

Safe to skip the first time: the prob­a­bil­ity in Sec­tion 3.3, the deriva­tions in Sec­tion 7.4, and Sec­tions 2.3, 2.6, 4.8, 5.4, 6.3⁠–⁠6.4, 9.5⁠–⁠9.6, 10.6⁠–⁠10.8, 11.5⁠–⁠11.6, 11.8⁠–⁠11.9, 12.7⁠–⁠12.9 and 13.5⁠–⁠13.9, and Chap­ter 14 unless you train or eval­u­ate agents.

By role, if you plan runs, read Chap­ters 1, 6, 11, 12 and 15 closely. If you post-train, read Chap­ters 8, 9, 10 and 14. If you serve models, read Chap­ters 1, 2 and 13.

DeepSeek-V3's post-train­ing used 5K of its 2,788K H800 GPU-hours. What share is that, and does a small share mean post-train­ing mat­ters little?

5 / 2,788 = 0.18% (com­puted). A small share does not mean a small effect. R1-Zero's RL used 101K GPU-hours, 101 / 2,788 = 3.6% of V3's total (com­puted), and took AIME 2024 pass@1 from 15.6% to 77.9% (reported). A com­pute share mea­sures cost, not effect, and the cheap stages are where much of a model's behav­ior is decided.

At $2 per GPU-hour, what would Llama 3.1 405B's 30.84M GPU-hours cost to rent, and what does that number leave out?

30.84M × $2 = $61.7M (com­puted, illus­tra­tive price). It leaves out the research and abla­tion runs before the final run, data col­lec­tion and human label­ing, much of the eval­u­a­tion, the people, and the cost of owning a clus­ter rather than rent­ing one. Those omis­sions can be large: SmolLM3's abla­tions and debug­ging alone came to 161,280 / 276,480 = 58.3% of its main run (com­puted from reported GPU-hours).

Why does the pipe­line figure draw eval­u­a­tion as a band over every stage rather than as a last step?

Because mea­sure­ments steer each stage. Small-model abla­tions choose the data, bench­mark scores choose anneals and check­points, each post-train­ing round is judged before the next, and RL rewards are them­selves auto­matic eval­u­a­tions. Eval­u­at­ing only at the end would find prob­lems after the com­pute is spent, and with­out error bars it could not tell a real gain from noise.

Llama 3 405B's train­ing time comes out as 66 days from its FLOPs and 78 days from its GPU-hours. Why do two honest cal­cu­la­tions dis­agree?

They mea­sure dif­fer­ent things. 66 days divides $3.8\times {10}^{25}$ FLOPs by 16,384 GPUs run­ning at 41% MFU, which is the steady-state pace of the main con­fig­u­ra­tion with no time lost. 78 days divides the model card's 30.84M GPU-hours, every hour charged to the model, by the same 16,384 GPUs (com­puted). The report does not give the full wall-clock time or split the gap, so use 66 days as the pace at full speed and 78 days as the all-in charge in clus­ter-days.

Why does this guide write Llama 3 8B's size as 8,030,261,248 param­e­ters?

The count is com­puted from the released con­fig­u­ra­tion, so every later for­mula uses the same exact number and results agree across chap­ters. "8B" is the rounded name. For exam­ple, at 16 bytes of model state per param­e­ter under mixed-pre­ci­sion AdamW, 16 × 8,030,261,248 = 128.5 GB (com­puted). Round­ing to 8B would give 128.0 GB. The 0.5 GB (0.4%) gap is small, but exact counts let every chap­ter's num­bers be checked against each other to the digit.

1. FLOPs and memory: the arithmetic of training

A train­ing plan begins with param­e­ters, tokens, FLOPs and bytes. Count them before book­ing the hard­ware, then turn the com­pute into time at a stated uti­liza­tion. Memory and band­width decide whether that arith­metic describes a run the machine can actu­ally exe­cute.

Why should a research engi­neer be able to do the arith­metic of train­ing by hand? A wrong guess costs months. Llama 3 405B has 405B train­able param­e­ters and was pre­trained on 15.6T text tokens with $3.8\times {10}^{25}$ FLOPs, on up to 16K H100 GPUs, each run­ning at 700 W with 80 GB of HBM3 (high-band­width memory, the GPU's main memory) [6]. At the 41% model FLOPs uti­liza­tion the paper reports on 16,384 GPUs, that is 66 days of unin­ter­rupted com­pute (com­puted in Sec­tion 1.3). The Llama 3.1 model card reports 30.84M H100 GPU-hours for 405B and 39.3M for all three sizes [11]. DeepSeek-V3 reports 2.788M H800 GPU-hours for its full train­ing [2].

Memory decides what fits. A 1.5B-param­e­ter GPT-2 needs 3 GB for its 16-bit weights, "yet, it cannot be trained on a single GPU with 32GB memory using Ten­sor­flow or PyTorch" [22]. EleutherAI is blunter: "Modern GPUs are typ­i­cally bot­tle­necked by memory, not FLOPs, for LLM train­ing." [23] The same guide warns that whitepa­per peak FLOP/s are never reached in prac­tice [23].

The order: param­e­ters (Sec­tion 1.1), FLOPs (1.2), days (1.3), train­ing bytes (1.4, 1.5), infer­ence bytes (1.6), a chip's speed limit (1.7), and energy and dol­lars (1.8). The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. Archi­tec­ture is Chap­ter 4, par­al­lel­ism Chap­ter 11, serv­ing Chap­ter 13.

1.1 Counting parameters

Exactly how many param­e­ters does a model have, and which of them do work in a matmul? A decoder-only trans­former is an embed­ding table that turns each token into a vector of width $d$, $L$ iden­ti­cal blocks, a final norm, and an output head that turns the last vector into $V$ vocab­u­lary scores. A Llama-style block uses grouped-query atten­tion (GQA, below), a SwiGLU MLP (three matri­ces, one of them a gate) and RMSNorm (nor­mal­iza­tion with one learned gain vector). If these are new, read Sec­tion 4.2 first: it walks one Llama 3 8B block oper­a­tion by oper­a­tion, and this sec­tion only counts its weights. Nearly all param­e­ters sit in the blocks' atten­tion and MLP weight matri­ces (Table 1.1). In Llama 3 8B the MLP holds 80.8% of each block (com­puted) [6, 17].

Out­side the blocks sit the input table ($Vd$), a final norm ($d$) and, for an untied head (its own matrix, not the input table reused), another $Vd$. With $[\text{untied}]$ equal to 1 or 0:

Kaplan et al.'s rule of thumb, $N\approx 12L{d}^2$, counts only the blocks, "where we have excluded biases and other sub-lead­ing terms" [25]. It assumes full multi-head atten­tion ($4d^2$) and a two-matrix $4d$ MLP ($8d^2$). Llama 3 8B's block is $13d^2$, so the rule gives 6.44B against 6.98B actual non-embed­ding param­e­ters (com­puted). SwiGLU's three matri­ces would nor­mally shrink the hidden width to about $8d/3$ [26], which OLMo 2 rounds up "to the clos­est mul­ti­ple of 128" [7], but Llama 3 uses $3.5d$ (8B, 70B) and $3.25d$ (405B). Grouped-query atten­tion (GQA) lets query heads share key and value heads, and Llama 3 uses 8 KV heads "to improve infer­ence speed and to reduce the size of key-value caches during decod­ing" [6]. That also cuts atten­tion to 41.9M param­e­ters per layer from 67.1M (com­puted).

Not every param­e­ter does a matmul: "embed­ding layers have a neg­li­gi­ble con­tri­bu­tion to the model FLOP counts, since they do not require matrix-vector prod­ucts" (Porian et al. [27]), while the output head is a real $d\times V$ matmul. gpt-oss agrees: "Unem­bed­ding param­e­ters are counted towards active, but not embed­dings." [28]. The two vocab­u­lary matri­ces are 13.1% of Llama 3 8B but 1.0% of 405B (com­puted), so total and non-embed­ding counts diverge most for small models, which mat­ters for scal­ing laws (Chap­ter 6). Shapes also serve the hard­ware: a vocab­u­lary divis­i­ble by 64 is rec­om­mended [29], and 128,256 is 64 × 2,004 (Sec­tion 1.7).

Common mis­takes: $V$ from the paper, a tied head treated as untied (GPT-2 ties it, Llama 3 and OLMo 2 do not [20, 17, 21]), missed extras such as OLMo 2's QK-norm [7], MoE totals used for FLOPs, and extra mod­ules: DeepSeek-V3's 685B release adds a 14B multi-token pre­dic­tion module to the 671B model [31]. Apply the for­mula to config.json, then sum numel() over the check­point. They should agree exactly.

1.2 FLOPs per token: the 6N rule and the attention term

Why does train­ing cost about $6N$ FLOPs per token, and when does atten­tion break that rule? A FLOP is one float­ing-point add or mul­ti­ply. FLOPs is a count and FLOP/s a rate: a chip deliv­ers TFLOP/s (${10}^{12}$ per second), a run con­sumes FLOPs. Older papers use the PF/s-day (writ­ten PF-day by Kaplan et al.), one PFLOP/s sus­tained for a day, $8.64\times {10}^{19}$ FLOPs [25].

From one matmul to one train­ing token

Mul­ti­ply­ing an $m\times k$ matrix by a $k\times n$ matrix gives $mn$ out­puts, each a length-$k$ dot prod­uct of $k$ mul­ti­plies and $k$ adds [32, 25]:

Llama 3 8B's query pro­jec­tion on 8,192 tokens costs

Through the whole net­work each weight does one mul­ti­ply and one add per token, so the for­ward pass costs $2N$ [33]. Back­ward needs two prod­ucts per for­ward one. For $Y=XW$, the input gra­di­ent $(\partial ℒ/\partial Y)W^{\top }$ keeps back­prop­a­ga­tion going and the weight gra­di­ent $X^{\top }(\partial ℒ/\partial Y)$ feeds the opti­mizer. Each matches the for­ward matmul, so back­ward costs $4N$ [34, 35], and GPT-3 adds "a mul­ti­plier of 3x to account for the back­wards pass" [36]. Train­ing costs about $6N$ FLOPs per token [25].

The atten­tion term

Atten­tion also has two mat­muls with no weights, the scores $Q{K}^{\top }$ and the weighted sum of values. For a query attend­ing to $\text{ctx}$ keys each costs $2\text{ctx}h{d}_h$ per layer, so the pair costs $4\text{ctx}h{d}_h$ for­ward and three times that in train­ing. Drop­ping the input table from $N$, since it does no matmul, gives

with $s$ the sequence length (keep $N$ whole for a tied head). The full count, used by PaLM [34] and nanoGPT [37], charges every query for every key. Under a causal mask token $i$ sees only $i$ keys, so FlashAttention-2 halves its count [38], and Porian et al. fault full counts for ignor­ing causal­ity [27]. Kaplan et al. left the con­text-depen­dent term out of $C$ as "a rel­a­tively small frac­tion of the total com­pute" for models with $d>s/12$ [25], and GPT-3 ignored atten­tion as under 10% [36]. For Llama 3 8B at 8K, causal atten­tion per token is

Table 1.4 repeats this for all three sizes. Atten­tion mat­ters more for small models, since its cost per layer grows with $d$ and the weights' with $d^2$. At 128K, $6N$ under­es­ti­mates the work 1.7 times (405B, causal) to 5.2 times (8B, full). With causal count­ing, atten­tion equals all other FLOPs at about 57K tokens for 8B, 106K for 70B and 196K for 405B (com­puted). How to Scale Your Model's rule (Austin et al., called the scal­ing book below) that atten­tion dom­i­nates once the con­text exceeds eight times the width [39] assumes other shapes, so com­pute the cross­over for yours. Figure 19 lets you slide the con­text and watch atten­tion's share grow.

So Llama 3 trains long sequences last: "We do not train on long sequences ear­lier because the com­pute in self-atten­tion layers grows quadrat­i­cally in the sequence length." [6] Its long-con­text stage used about 800B tokens [6] (Chap­ter 7).

Total com­pute: does 6ND match the papers?

Mul­ti­ply­ing by $D$ tokens gives the budget that Chin­chilla's scal­ing anal­y­sis uses as its con­straint [40]:

Table 1.5's matches look per­fect, but many reported totals are them­selves 6ND. GPT-3's appen­dix mul­ti­plies param­e­ters by tokens and ignores atten­tion [36], and OLMo 2 com­putes train­ing FLOPs "using the approx­i­ma­tion from Kaplan et al. (2020)" [7]. Others count more: PaLM's total adds atten­tion, 3.28 TFLOPs per token against 3.24 for $6N$ [34], and Chin­chilla's detailed count dif­fers from 6ND by 0.99 to 1.10 times across model sizes [40]. A match with 6ND checks the inputs, not the method.

Common mis­takes: a mul­ti­ply-add counted as one FLOP, the input table kept in $N$ (7.0% too high for 8B, com­puted), MoE totals instead of acti­vated param­e­ters, atten­tion ignored at long con­text, causal and full counts mixed, recom­pu­ta­tion counted as model FLOPs. To plan, use $6N_{\text{matmul}}D$ plus causal atten­tion, with $N_{\text{matmul}}$ the weights that do mat­muls (acti­vated for an MoE, head in, input table out). To com­pare with papers, use plain 6ND, as most reports do.

1.3 From FLOPs to days: utilization, time to train and GPU-hours

How long will a run take on a given clus­ter, and what does MFU really mea­sure? Time is work over rate. With $p$ accel­er­a­tors of peak $F$ FLOP/s run­ning at model FLOPs uti­liza­tion (MFU, the frac­tion of peak spent on the model's own FLOPs),

GPU-hours do not depend on $p$ while MFU holds. EleutherAI writes $C\approx \tau T=6ND$, with $\tau$ the clus­ter's through­put [23]. Mega­tron's GPT-3 175B esti­mate checks out: 300B tokens on 1,024 A100s at 140 TFLOP/s each, with $8N$ per token under full recom­pu­ta­tion, take

against the paper's "34 days" [41].

MFU, HFU and recom­pu­ta­tion

PaLM defined MFU as "the ratio of the observed through­put (tokens-per-second) rel­a­tive to the the­o­ret­i­cal max­i­mum through­put of a system oper­at­ing at peak FLOPs" [34], with the max­i­mum count­ing only the model's for­ward and back­ward FLOPs:

Hard­ware FLOPs uti­liza­tion (HFU) also counts recom­pu­ta­tion, which MFU leaves out [44]. Recom­pu­ta­tion (Sec­tion 1.5) reruns the for­ward pass in back­ward to save memory, so the full ver­sion exe­cutes $2N+2N+4N=8N$ per token [39] and HFU is 1.33 times MFU. With a frac­tion $\rho$ of the for­ward pass recom­puted,

It costs time: Kor­thikanti et al. saw 30⁠–⁠40% longer steps with full recom­pu­ta­tion, and selec­tive recom­pu­ta­tion with sequence par­al­lel­ism lifted a 530B model from 42.1% to 54.2% MFU [45]. PaLM rejects HFU as "not a con­sis­tent and mean­ing­ful metric" [34]: recom­put­ing more raises it with­out train­ing faster.

Why do the best runs stop near 40⁠–⁠60%? Losses stack, and no source splits them the same way. Norms, soft­max and rout­ing take time but add no counted FLOPs, and MoE adds "many memory-inten­sive oper­a­tions like rout­ing, local scat­ter, and gather" [50]. DeepSeek-V3 gives 20 of the H800's 132 stream­ing mul­ti­pro­ces­sors (SMs, Sec­tion 2.1) to com­mu­ni­ca­tion [2] and loses 2.06 s of a 19.9 s step to a pipe­line bubble, idle time while pipe­line stages wait on each other (Sec­tion 11.4) [1]. Long con­text, awk­ward shapes (Sec­tion 1.7) and memory pres­sure (Sec­tion 1.5) add more. Plan dense BF16 runs on H100s at 35⁠–⁠45% MFU [6, 48, 12, 1].

To com­pare reported hours with an MFU, use the same FLOP count:

with $N-Vd$ the matmul param­e­ters of Sec­tion 1.2 (keep $N$ whole for a tied head).

Fail­ure rates, the 1/N law and Llama 3's 466 inter­rup­tions are in Sec­tion 12.1. Good­put and ETTR (effec­tive train­ing time ratio) are defined in Sec­tion 12.8. In Figure 3, pick a run­ning exam­ple and com­pare the bar for hours at your MFU with the reported hours. The cal­cu­la­tor counts causal atten­tion by default, so Llama 3 405B shows 69 days. Set Count­ing to None (6N) to recover the 66 days above: part of every gap is the count­ing con­ven­tion.

The for­mu­las work at every scale. GPT-2 124M on 10B tokens is $8.55\times {10}^{18}$ FLOPs (nanoGPT's count), 95 min­utes on 8 A100s at 60% MFU (com­puted), near the 90 min­utes llm.c's author reports at up to about 60% [15]. An 8B model on 15T tokens on 1,024 H100s at 40% needs $7.23\times {10}^{23}$ FLOPs, 20.6 days and 507K GPU-hours (illus­tra­tive clus­ter, com­puted), a third of the 1.46M hours reported for Llama 3.1 8B.

Two cau­tions. Reported GPU-hours cover the final run only (Sec­tion 0.2). And MFU falls as GPUs are added at a fixed global batch. MegaScale went from 59.1% on 3,072 GPUs to 55.2% on 12,288 [47], and Llama 3 from 43% to 41% on twice the GPUs "due to the lower batch size per DP group needed to keep the global tokens per batch con­stant during train­ing" [6]. Opti­miza­tion fixes the batch (Chap­ter 5), so each GPU gets less work while com­mu­ni­ca­tion stays fixed. Twice the clus­ter does not halve the time.

1.4 Training memory I: weights, gradients and optimizer states

How many bytes per param­e­ter does train­ing need, and how does shard­ing divide them? Train­ing keeps 16-bit weights, their gra­di­ents and the opti­mizer's pri­vate state for every param­e­ter. ZeRO calls these the model states and every­thing else (acti­va­tions, tem­po­rary buf­fers, frag­mented memory) the resid­ual states [22]. This sec­tion counts model states. Sec­tion 1.5 counts the rest.

Six­teen bytes per param­e­ter

Take mixed-pre­ci­sion Adam, the default for large models. Mixed pre­ci­sion means the mat­muls run in a 16-bit format (BF16 or FP16) while the opti­mizer works in FP32. Each param­e­ter costs 2 bytes of 16-bit weight, 2 bytes of 16-bit gra­di­ent and three FP32 opti­mizer num­bers: a master copy of the weight, Adam's momen­tum and its vari­ance, 4 bytes each [22]. With ZeRO's $w$, $g$ and $K$ for weight, gra­di­ent and opti­mizer bytes per param­e­ter,

The master copy exists because updates are tiny next to the weights. FP16’s small­est pos­i­tive sub­nor­mal is $2^{-24}$; with round-to-near­est-even, mag­ni­tudes at or below half that value (2⁻²⁵) round to zero. Micike­vi­cius et al. keep an FP32 master copy so that small updates can accu­mu­late [51]. BF16 has FP32's range but far fewer man­tissa bits [52], so small updates round away (Chap­ter 5).

Accu­mu­lat­ing gra­di­ents over micro-batches in FP32 adds 4 bytes, making 20 per param­e­ter [44]. Llama 3 does this [6]. Do the sources dis­agree? Micike­vi­cius et al. found mixed pre­ci­sion roughly halved train­ing memory [51], while the Ultra-Scale Play­book says it saves none and only redis­tributes it [44]. Both are right. FP32 Adam also needs 16 bytes per param­e­ter, so model states do not shrink. The 2017 saving came from acti­va­tions.

Newer work goes lower (as of Novem­ber 2026): 4-bit states [55], GaLore's low-rank gra­di­ents [56], FP8 train­ing [57, 58] and no FP32 master weights at all [59]. Quan­tized states can go stale, though [60], and Chap­ter 5 judges whether they con­verge as well as Adam.

The Play­book rounds the 70B row to 1,120 and 1,400 GB [44]. The rule of thumb: an 80 GB GPU holds the model states of at most 80 / 16 = 5B param­e­ters, with noth­ing left for acti­va­tions (com­puted). Llama 3 8B cannot train with plain Adam on one H100.

Divid­ing model states across GPUs

Plain data par­al­lel­ism copies all $16N$ bytes to every GPU. ZeRO removes copies across the $p_{\text{dp}}$ data-par­al­lel GPUs in three stages: stage 1 shards opti­mizer states, stage 2 also gra­di­ents, stage 3 also weights [22]. Per GPU,

These repro­duce the paper's 7.5B model on 64 GPUs: 120, 31.4, 16.6 and 1.9 GB [22]. ZeRO's three stages and FSDP are in Sec­tion 11.2. Here we only need the bytes.

Llama 3's FSDP shards opti­mizer states and gra­di­ents but keeps gath­ered weights after the for­ward pass, to save an all-gather in the back­ward pass [6]. Offload­ing to CPU memory and NVMe goes fur­ther [61, 62], and DeepSeek-V3 keeps its moving aver­age of the weights in CPU memory [2]. LoRA and QLoRA memory and FLOPs are in Sec­tion 8.3.

In Figure 4, pick Llama 3 70B, set $p_{\text{dp}}$ to 64 and step ZeRO from 0 to 3: opti­mizer states col­lapse at stage 1, gra­di­ents at 2, weights only at 3. Then raise the sequence length and watch acti­va­tions decide whether the bar crosses the HBM line.

The usual mis­takes: for­get­ting the FP32 gra­di­ent buffer, count­ing the master copy twice (it is inside $K$), and test­ing only step one. Runs can pass step one and fail on step two, because the opti­mizer state builds up after the first step [44].

1.5 Training memory II: activations, recomputation and the real peak

How much memory do acti­va­tions take, what does recom­pu­ta­tion buy, and why does a run that fits on paper still run out of memory? The acti­va­tions are the ten­sors the for­ward pass saves for the back­ward pass. They grow with the tokens on the GPU, the layers and the width. They pile up through the for­ward pass, peak at the loss and are freed layer by layer on the way back. ZeRO does not shrink them.

Count­ing saved ten­sors

Kor­thikanti et al. did the book­keep­ing for a GPT layer with 16-bit acti­va­tions, multi-head atten­tion, a 4d GeLU MLP and dropout. In our nota­tion (their $h$ is our width $d$, their $a$ our head count $h$), $b$ sequences of $s$ tokens store $11sbd+5h{s}^{2}b$ bytes in atten­tion, $19sbd$ in the MLP and $4sbd$ in the two layer norms [45]:

The $5hs/d$ term holds the atten­tion scores, soft­max and dropout mask, the only ten­sors that grow with $s^2$. For GPT-3 ($h=96$, $s=\mathrm{2,048}$, $d=\mathrm{12,288}$) it equals 80, as the paper states [45], so one sequence needs 2.67 GiB per layer (com­puted). That qua­dratic part is cheap to recom­pute, and recom­put­ing only it saves 70% and 65% of acti­va­tion memory for GPT-3 and MT-NLG at 2.7% and 1.6% extra FLOPs [45]. FlashAttention never stores the $s\times s$ matrix and recom­putes it in the back­ward pass, so it runs faster and uses less memory [63], linear in sequence length. FlashAttention's tiling and online soft­max are explained in Sec­tion 2.4.

A Llama layer

No pub­lished count covers a Llama layer (GQA, SwiGLU, RMSNorm, no dropout), so we do our own. With FlashAttention, an unfused SwiGLU and BF16, each token in each layer saves the input and output of both RMSNorms ($4d$ ele­ments), the query and atten­tion output ($2h{d}_h$), the key and value ($2h_{kv}{d}_h$), and four MLP ten­sors of width $d_{\text{ff}}$ (gate output, its SiLU, up output and their prod­uct):

For Llama 3 8B that is $2\times (\mathrm{16,384}+\mathrm{8,192}+\mathrm{2,048}+\mathrm{57,344})=\mathrm{167,936}$ bytes, or $41.0d$, per token per layer. One 8,192-token sequence through 32 layers holds 41.0 GiB (44.0 GB). Llama 3 70B holds 202.5 GiB and 405B 602.4 GiB per 8K sequence (all com­puted). Treat these as esti­mates: on a scaled-down Llama-shaped layer, the ten­sors PyTorch auto­grad saved came to 8% more (mea­sured for this guide), from K and V expanded to every query head, the soft­max log-sum-exp and the norm sta­tis­tics. With­out FlashAttention the stored soft­max adds $2h{s}^{2}b$ bytes per layer, 128 GiB more for 8B at 8K (com­puted).

Recom­pu­ta­tion

Acti­va­tion recom­pu­ta­tion (or check­point­ing) trades FLOPs for bytes: store less in the for­ward pass and recom­pute the rest in the back­ward pass. Full recom­pu­ta­tion stores only each layer's input, $2sbdL$ bytes [45], and reruns the for­ward pass: about $8N$ hard­ware FLOPs per token instead of $6N$, which is why HFU exceeds MFU (Sec­tion 1.3). Chen et al. (2016) made it prac­ti­cal for deep net­works: check­point­ing an $n$-layer net­work every $n^{1/2}$ layers needs memory that grows as $n^{1/2}$, for one extra for­ward com­pu­ta­tion [64].

Kor­thikanti et al. mea­sured a per-layer over­head of 7% for selec­tive against 39% for full recom­pu­ta­tion, and 30⁠–⁠40% more exe­cu­tion time in full-recom­pu­ta­tion runs [45]. Llama 3 pre­trained on 8K sequences with­out acti­va­tion check­point­ing [6]. Sec­tion 11.4 set­tles whether Llama 3 405B fits at 8K with­out check­point­ing (about 42⁠–⁠75 GiB per GPU, depend­ing on assump­tions).

What par­al­lel­ism divides

Tensor par­al­lel­ism with sequence par­al­lel­ism (Sec­tion 11.3) splits acti­va­tions across $p_{\text{tp}}$ GPUs: Llama 3 70B at 8K drops from 202.5 to 25.3 GiB per GPU at $p_{\text{tp}}=8$ (com­puted). Pipe­line par­al­lel­ism does not help the first stage: under 1F1B (the one-for­ward-one-back­ward sched­ule of Sec­tion 11.4) it holds $p_{\text{pp}}$ micro-batches of $L/p_{\text{pp}}$ layers, $L$ layers' worth, and more with $v$ inter­leaved chunks [45]:

which is $1.47L$ at $p_{\text{pp}}=16$, $v=2$ (com­puted). Llama 3's first stage used more memory [6]. Chap­ter 11 divides these num­bers for each layout.

The logits

The loss needs $sbV$ logits, in BF16 and again in FP32 [45], so $M_{\text{logits}}=2sbV+4sbV$ bytes. With Llama 3's 128,256-token vocab­u­lary that is 1.96 + 3.91 = 5.9 GiB (6.3 GB) at 8K and 94 GiB at 128K (com­puted). Fused loss ker­nels remove most of it: Cut Cross-Entropy cuts Gemma 2 2B's loss memory from 24 GB to 1 MB [65], and the Liger ker­nels report 60% less GPU memory [66].

Figure 5 walks through that step. Watch acti­va­tions climb about 1.4 GB per layer, the logits spike at the loss, and the back­ward pass free them. Switch to step 1 to see the 8 GB of Adam moments appear at the first opti­mizer step.

1.6 The arithmetic of inference: FLOPs per token, weights and the KV cache

What does gen­er­at­ing one token cost in FLOPs and bytes, and how big is the KV cache? With no back­ward pass and no opti­mizer, the $6N$ FLOPs per train­ing token become $2N$ per infer­ence token [70, 33], and model states shrink to the weights. A new memory takes their place: the KV cache, the keys and values of every ear­lier token, kept so they are not recom­puted for each new one.

FLOPs per token

Pre­fill pro­cesses the prompt in one for­ward pass, at $2N$ per prompt token plus causal atten­tion. Decode then gen­er­ates one token at a time. Each new token passes through every weight once, and each of the $h$ query heads in each layer takes a dot prod­uct with $\text{ctx}$ cached keys and a weighted sum of $\text{ctx}$ values, $2d_h\text{ctx}$ FLOPs each:

We sub­tract the $Vd$ input-embed­ding param­e­ters because a lookup is not a matmul. Snell et al. use the same $2N$ per token to com­pare test-time with pre­train­ing com­pute [71]. With the Llama 3 shapes [6]:

At 128K a decoded token costs 2.3 times (405B) to 5.6 times (8B) its weight FLOPs (com­puted), so $2N$ fails worst for small models with long con­texts.

The KV cache

Each layer caches one key and one value vector per KV head per token. With $e$ bytes per ele­ment,

Check it on OPT-13B, with 40 layers, width 5,120 and multi-head atten­tion ($h_{kv}{d}_h=d$): $2\times 40\times \mathrm{5,120}\times 2=\mathrm{819,200}$ bytes, which the vLLM paper rounds to 800 KB [73].

Sec­tion 4.5 com­pares KV bytes per token across atten­tion designs: 327,680 B for Llama 3 70B with GQA-8 against 70,272 B for DeepSeek-V3 with MLA. Both descend from multi-query atten­tion, which shares one key and value head across all heads to cut the memory band­width of decod­ing [74, 75].

The cache mat­ters more than its FLOPs sug­gest because, unlike the weights, it is unique to each sequence [33]. Weight load­ing dom­i­nates at small batches and short sequences, KV load­ing at 2,048+ tokens with batch 512+, where a 500B+ multi-head model needs a 3 TB cache [33].

Weights and what fits

Llama 3 70B in BF16 is 141.1 GB, more than one H100. Llama 3 405B in BF16 is 811.7 GB (com­puted) and does not fit one 8-GPU H100 machine [6]. FP8 halves weight memory [1], to 405.9 GB (com­puted). Sec­tion 13.2 owns the KV-cache arith­metic, the decode floor and what fits in HBM. Other infer­ence over­head is usu­ally 20% or less [23], but the cache used to waste space: before paging, only 20.4⁠–⁠38.2% of KV memory held token states, and vLLM's paged cache raised through­put 2⁠–⁠4 times [73]. Sec­tion 13.4 works out how many sequences fit: on 8 H100s a 402.9 GB pool holds 1.23M tokens, about 150 sequences that each reserve 8K.

Decode does about 2 FLOPs per weight per sequence and a few per cached byte. With so few FLOPs per byte, memory band­width rather than peak FLOP/s sets decode speed. Sec­tion 1.7 makes that pre­cise with the roofline.

1.7 Rooflines: when bytes, not FLOPs, set the time

Given an oper­a­tion's FLOPs and bytes, how fast can it run on a given chip? Think of the chip as a fac­tory with a load­ing dock [77]. The floor (the tensor cores) does $F_{\text{peak}}$ FLOPs per second. Mate­rial arrives through the dock (HBM) at $W$ bytes per second. If each byte feeds few oper­a­tions, the floor idles.

The roofline and the ridge point

An oper­a­tion that does $n_f$ FLOPs and moves $n_b$ bytes needs at least $n_f/F_{\text{peak}}$ sec­onds of math and $n_b/W$ sec­onds of traf­fic. With per­fect over­lap the time is the larger one, so

This is the roofline of Williams, Water­man and Pat­ter­son [78, 79]. $I=n_f/n_b$ is the arith­metic inten­sity in FLOPs per byte of HBM traf­fic (the paper says oper­a­tional inten­sity). The ridge point $I^{*}$ is "the min­i­mum oper­a­tional inten­sity required to achieve max­i­mum per­for­mance" [79], which NVIDIA calls the ops:byte ratio [80]. Below it an oper­a­tion is memory-bound, above it com­pute-bound. The roof is a best case: microbench­marks reach 90⁠–⁠92% of rated band­width [81], and an H100 does 989 TFLOP/s of FP16 matmul but only 3.9 TFLOP/s of expo­nen­tials [82].

Ridges clus­ter at 150⁠–⁠300 FLOP per byte in BF16 and double in FP8, because com­pute out­runs band­width: over 20 years peak server FLOP/s grew 60,000x against 100x for DRAM band­width [89].

Where trans­former oper­a­tions sit

An $(M\times K)(K\times N)$ matmul (GEMM) costs $2MNK$ FLOPs, the rule of Sec­tion 1.2 (here $N$ is the output width, not the param­e­ter count). With each oper­and cross­ing HBM once at $e$ bytes per ele­ment [32],

where $T$ is the tokens in the prod­uct. In BF16 a weight GEMM's inten­sity is about the tokens in flight, and a matrix-vector prod­uct is always memory-bound [32].

Decode atten­tion cannot be batched up the slope. Each sequence reads its own cache, and the $g=h/h_{kv}$ query heads shar­ing a KV head do $4g{d}_h$ FLOPs per cached posi­tion:

4 FLOP per byte for Llama 3 8B and 8 for 70B at any batch or con­text (com­puted). Only fewer KV bytes help (Sec­tion 1.6). A roofline study of Llama-2⁠–⁠7B found that "in the decode stage, all com­pu­ta­tions are memory-bound" [90]. Ele­men­t­wise oper­a­tions do about one FLOP per two-byte ele­ment [80]. In a Llama 3 8B layer at 8K tokens they take about 10% of ideal H100 time for 0.03% of the FLOPs (com­puted). Fusion, which keeps inter­me­di­ates on chip, is the fix [77].

Figure 6 places these oper­a­tions on real chips. Drag the token count: the four weight GEMMs climb the slope and cross the H100 ridge at $T\approx 308$–345 (com­puted), a little above 295 because acti­va­tions add traf­fic, while decode atten­tion stays pinned at 4 or 8. Com­pare the H200's line: the same roof, with the ridge moved left to 206.

Decode against the memory wall

A decode step with batch $b$ reads every weight once for $2Nb$ FLOPs, so $I_{\text{decode-weights}}\approx 2Nb/(e_{w}N)=2b/e_w$, about $b$ with BF16 weights.

PaLM 540B showed both regimes on 64 TPU v4 chips: 29 ms per token in low-batch gen­er­a­tion with int8 weights and 76% MFU on large batches of input tokens [33]. Serv­ing sys­tems now split the two phases [92], as Chap­ter 13 shows. Mix­ture-of-experts decode is harder: DeepSeek-V3 routes each token to 8 of 256 experts, so feed­ing every expert near the ridge takes about 9,450 tokens per step (com­puted), and its small­est decod­ing deploy­ment is 320 GPUs [2].

Over­head and shapes

Horace He adds a third regime, over­head: Python, dis­patch and kernel launches [77]. His test: "if you double your batch size but your run­time only increases by 10%, you're likely over­head bound" [77].

Shapes matter too: on a 108-SM A100, a matmul need­ing 117 tiles instead of 108 adds a tail wave and roughly halves through­put [32]. The expo­nen­tial unit is a ceil­ing of its own: FlashAttention-3 lifted atten­tion from 35% to 75% of H100 peak by over­lap­ping expo­nen­tials with mat­muls [82], and Sec­tion 2.4 explains its tiling and online soft­max. To diag­nose a kernel, count $n_f$ and $n_b$, com­pute both time bounds, and if the mea­sured time is far above the larger one, look for over­head, bad shapes or a lower ceil­ing.

1.8 Energy, dollars and the compute trend

What does a train­ing run cost in energy and money, and how fast is train­ing com­pute grow­ing? Both costs start from the GPU-hours of Sec­tion 1.3:

Power usage effec­tive­ness (PUE) is facil­ity energy over IT energy: 1.58 for the 2018 US aver­age [93] and 1.09 across Google's fleet [94]. $ϵ$ is energy per model FLOP.

The card assumes full rated power through­out [95]. OLMo 2's mea­sured power swung between about 15% and 85% of max­i­mum, just over 100 W per GPU during check­points [95], so TDP-based esti­mates "are nearly always an over­es­ti­ma­tion" [96]. GPU-only bound­aries err the other way: one median Gemini prompt is 0.24 Wh with host and facil­ity counted and 0.10 Wh with accel­er­a­tors only [94]. Always state the bound­ary.

Facil­ity power and the grid swings of syn­chro­nous jobs are cov­ered in Sec­tion 2.7.

Dol­lars: the method decides the number

DeepSeek-V3's famous figure is 2.788M H800-hours at an assumed 2 dol­lars per GPU-hour, 5.576 mil­lion dol­lars, final run only, exclud­ing prior research and abla­tions [2]. The same model can carry very dif­fer­ent prices.

Amor­tized fron­tier train­ing costs grew 2.4x per year since 2016. Accel­er­a­tor chips are 44% of amor­tized hard­ware and energy cost and energy about 9%, and energy falls to 2⁠–⁠6% of the full devel­op­ment cost once staff are counted [100].

The com­pute trend

Epoch AI esti­mates that train­ing com­pute grew 4.1x per year for notable models from 2010 to May 2024 and 5.3x for fron­tier models, slow­ing to 4.2x after 2018, and sum­ma­rizes the trend as 4⁠–⁠5x per year [101]. Our least-squares fit of ${\log }_{10}C=a+kt$ on Epoch's Novem­ber 2026 data gives ${10}^k=5.47$x per year (90% inter­val 4.9⁠–⁠6.1x) for 302 lan­guage models since 2010, a dou­bling every 4.9 months, and 4.75x (4.3⁠–⁠5.2x) for 45 fron­tier models since 2018 (com­puted from [98]).

Figure 7 plots that dataset. Find the run­ning exam­ples on the trend, then note the hollow points above ${10}^{26}$ FLOPs: Epoch's esti­mates for closed models, labeled Likely or Spec­u­la­tive, not lab-reported num­bers.

Algo­rithms add more. The com­pute needed for a fixed lan­guage-model loss has halved about every 8 months (95% inter­val 5⁠–⁠14) [102], so effec­tive com­pute grows roughly 10⁠–⁠15x per year (illus­tra­tive: 4⁠–⁠5x times about 2.8x). Epoch judges $2\times {10}^{29}$ FLOP runs likely fea­si­ble by 2030 [103]. Chap­ter 6 asks what that com­pute buys.

Llama 3 8B trains at 8K tokens, then at 128K. How much more does a train­ing token cost at 128K, and why?

About 2.9x: 148 against 51 GFLOPs per token, causal count­ing (com­puted from [6]). Weight FLOPs stay near 45 GFLOPs. The atten­tion mat­muls grow lin­early with con­text and become about 70% of the work.

PaLM reports 46.2% MFU and 57.8% HFU. What recom­pu­ta­tion does the ratio imply?

57.8 / 46.2 = 1.25 = $7.5N/6N$: $1.5N$ extra FLOPs per token, 75% of the for­ward pass. PaLM recom­puted 75% of its non-atten­tion for­ward FLOPs [34]. Full recom­pu­ta­tion would give 1.33.

Llama 3 70B, mixed-pre­ci­sion AdamW, 64 GPUs: per-GPU model states with­out shard­ing and with ZeRO stages 1, 2 and 3?

About 1,129, 295, 157 and 17.6 GB (com­puted with the ZeRO for­mu­las [22]). Only stage 3 fits an 80 GB GPU, because stages 1 and 2 keep all 141 GB of BF16 weights every­where.

GQA with 8 KV heads cuts Llama 3 70B's KV cache 8x. Does it cut decode atten­tion FLOPs 8x?

No. All 64 query heads still score every cached key, so FLOPs follow $h$, not $h_{kv}$ [75]. GQA cuts bytes 8x, which is why decode atten­tion runs at 8 FLOP per byte.

An H100 sheet lists 1,979 TFLOP/s BF16 and 3.35 TB/s. What is the dense ridge, and what decode batch makes weight GEMMs com­pute-bound?

1,979 includes spar­sity [4], so dense is 989.5 and the ridge 295 FLOP per byte. BF16 decode does 2b FLOPs per 2 bytes of weights, so the cross­over is near b = 295. The sparse figure gives 591, a 2x error.

2. Hardware: accelerators, interconnects and clusters

A chip's peak is only one part of its per­for­mance. Follow the bytes from reg­is­ters and high-band­width memory through the node and across the clus­ter: the slow­est nec­es­sary trans­fer can set the pace for every­thing else.

Llama 3 405B trained on up to 16,384 H100 GPUs at 380 to 430 TFLOP/s per GPU, a BF16 MFU of 38% to 43% [6] (MFU is defined in Sec­tion 1.3). So the tensor cores, with a dense peak of 989.5 TFLOP/s, deliv­ered less than half of it as useful model FLOPs (com­puted, 380/989.5 and 430/989.5). Hard­ware also shapes models: DeepSeek-V3 trained on 2,048 H800 GPUs [2], whose NVLink "is reduced from 900 GB/s to 400 GB/s", and so "Tensor Par­al­lel­ism is avoided during train­ing" [1].

On an H100, band­width falls roughly ten­fold at each step out from the tensor cores: about 31 TB/s from shared memory, 3.35 TB/s from HBM, 450 GB/s per direc­tion over NVLink, 50 GB/s through a 400 Gb/s net­work card and about 7 GB/s per GPU across the top of Llama 3's net­work [82, 4, 104, 6]. Ker­nels, for­mats, par­al­lel­ism and net­works all keep traf­fic on the fastest level that can hold it.

Sec­tions 2.1 to 2.4 stay inside one chip. Sec­tion 2.5 to Sec­tion 2.8 move out to the node, net­work, clus­ter and chip land­scape. The H100 SXM (the sock­eted server module, not the PCIe card) is the ref­er­ence chip, with specs as of Novem­ber 2026. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

2.1 Inside a GPU: SMs, warps and tensor cores

When PyTorch mul­ti­plies two matri­ces, which hard­ware does the work, and where does the peak FLOP/s on the spec sheet come from? The tensor cores. "An H100 GPU has 989 TFLOPs of half-pre­ci­sion matrix mul­ti­ply com­pute, and ~60 TFLOPs of 'every­thing else'" [105], so tensor cores supply about 94% of the arith­metic (com­puted).

A GPU is many copies of one pro­ces­sor, the stream­ing mul­ti­pro­ces­sor (SM) [106]. The H100 SXM has 132 SMs with 4 tensor cores each, cut down from 144 SMs on the full die [104]. A kernel launches thread blocks, each on one SM, whose threads form "groups of 32 threads called warps" [106] that exe­cute one instruc­tion together. Hopper adds warp­groups of four warps that issue matrix instruc­tions together [82, 81].

Loads take hun­dreds of cycles, so an SM keeps up to 64 warps res­i­dent and switches among them. Occu­pancy, res­i­dent warps over 64, is capped by the first resource to run out: 64K 32-bit reg­is­ters, 255 per thread, 32 blocks or 228 KB of shared memory per SM [107]. With $R$ reg­is­ters per thread,

so 32 reg­is­ters allow 100%, 128 allow 25% and 255 allow 12.5% (com­puted). Fast ker­nels accept low occu­pancy, because "reg­is­ters are gen­er­ally the key resource" [105]. In FlashAttention-3, pro­ducer warps only move data and con­sumer warp­groups only com­pute, with reg­is­ters shifted between them [82]: two con­sumers at 240 reg­is­ters per thread and a pro­ducer at 24 fill 64,512 of 65,536 reg­is­ters at 19% occu­pancy (illus­tra­tive, com­puted).

Peak FLOP/s is a prod­uct of three num­bers, with one fused mul­ti­ply-add (FMA) counted as 2 FLOPs:

Dense 16-bit matmul work per SM per clock has dou­bled each gen­er­a­tion, from 1,024 on V100 (FP16, since Volta has no BF16) to 2,048 on A100, 4,096 on H100 and 8,192 on B200 (BF16) [108, 109, 110]. Other chips follow the same for­mula. A TPU's matrix unit (MXU) is a sys­tolic array that passes oper­ands from cell to cell, 16K mul­ti­ply-accu­mu­lates per cycle on a 128 × 128 array, accu­mu­lated in FP32 [111]. Table 2.1 mul­ti­plies it out.

Non-matmul work is slower. An A100 has "312 TFLOPs/s of FP16/BF16 matmul, but only 19.5 TFLOPs/s of non-matmul FP32" [38]. Expo­nen­tials go to the multi-func­tion unit (MUFU) at 16 per clock per SM on H100 and B200 [110]: 16 × 132 × 1.83 GHz = 3.9 tril­lion per second, 256 times below the H100's BF16 matmul rate (com­puted). This unit shapes atten­tion ker­nels (Sec­tion 2.4).

2.2 The memory hierarchy: why moving bytes sets the pace

Where do the oper­ands of a matrix mul­ti­ply come from, how fast can each level deliver them, and why does data move­ment, not arith­metic, usu­ally set the speed? Memory gets smaller and faster closer to the tensor cores, from HBM (high-band­width memory) beside the die to an L2 cache, per-SM shared memory and reg­is­ters. For two decades peak server FLOP/s grew about 3.0 times every two years while DRAM and inter­con­nect band­width "have only scaled at 1.6 and 1.4 times every 2 years, respec­tively" [89].

Table 2.2 gives the H100's ladder. Latency grows down­ward, from 29 cycles for shared memory to 479 for HBM, mea­sured by Luo et al. on an H800 PCIe [81], about 17 and 273 ns (com­puted at 1,755 MHz).

The roofline and the ridge point are derived in Sec­tion 1.7. Apply them here with­out re-deriv­ing. The new point is that every level has its own ridge, $I^{*}=F/W$ [79]. For an H100 in BF16 it is 295 FLOPs per byte against HBM, 82 against L2 and 32 against shared memory, and FP8 dou­bles all three (com­puted). A kernel must reuse each HBM byte about 295 times to keep the tensor cores busy.

Figure 8 draws the chip beside this ladder. Switch chip and format to move each level's ridge point, and pick a kernel to see where it is com­pute-bound.

Black­well shows the pres­sure. B200 dou­bled the tensor rate, but for shared memory "the read through­put is 128 bytes / clock / SM, the same as Hopper", so that ridge dou­bles to 64. In atten­tion, shared-memory traf­fic and expo­nen­tials now exceed matmul time "by 25⁠–⁠60%" [110]. Black­well's answer is a new level, tensor memory (TMEM), 256 KB per SM for tensor-core results [110].

Latency mat­ters too. By Little's law, sus­tain­ing band­width $W$ at latency $\alpha$ needs $W\alpha$ bytes in flight: 3.35 TB/s × 273 ns = 0.91 MB, about 6.8 KiB per SM (illus­tra­tive, latency from another card). So Hopper added the Tensor Memory Accel­er­a­tor (TMA), a copy engine driven by a descrip­tor of "tensor dimen­sions and block coor­di­nates instead of per-ele­ment address­ing" [104], which kernel authors call "com­pletely indis­pens­able" [105].

Rivals bet dif­fer­ently. TPU v4 replaces caches with soft­ware-man­aged scratch­pads, 16 MiB per TensorCore plus a shared 128 MiB [112]. AMD's MI300X puts a 256 MB Infin­ity Cache in front of 192 GB of HBM3 at 5.3 TB/s [115, 113]. Common fail­ures are unco­a­lesced loads, where a warp's threads touch scat­tered addresses, and bank con­flicts, since shared memory is "banked" into 32 stores that seri­al­ize clash­ing accesses [105].

2.3 Number formats: from FP32 to FP4

What exactly is in the bits of each train­ing format, and why do train­ing recipes use the ones they do? Fewer bits buy band­width and through­put: "FP8 halves data stor­age require­ments and dou­bles through­put com­pared to FP16 or BF16" [104], so an H100's 989.5 dense TFLOP/s in BF16 become 1,979 in FP8 [4]. The price is range and pre­ci­sion. A normal float with expo­nent field $E$ and $n_m$ man­tissa bits stores

When $E=0$ the value is sub­nor­mal, $2^{1-\text{bias}}\times \text{frac}/2^{n_m}$ [116, 117]. Pre­ci­sion is the spac­ing $ϵ$ in $[1,2)$, and range is counted in powers of two, called binades (Table 2.3).

Limits are com­puted from the bit lay­outs [117, 116]. Figure 9 encodes any value in every format: type pi, 1,000 and $3\times {10}^{-5}$, see which rows round, sat­u­rate or flush to zero, then rescue the small value with the scale con­trol.

BF16 came from the TPU, where "for the first time the expo­nent (8 bits) is larger than its frac­tion (7 bits)" [118]. Keep­ing FP32's range "is impor­tant to ensure that no hyper-param­e­ter tuning is required for con­ver­gence" [52]. FP16 tops out at 65,504 and needs FP32 master weights, loss scal­ing and FP32 accu­mu­la­tion [51]. TF32 is a math mode, not a stor­age format [109].

FP8 has two encod­ings. E5M2 fol­lows IEEE con­ven­tions and reaches 57,344 over 32 binades. E4M3 gives up infini­ties and keeps one NaN pat­tern to reach 448 over 18 binades, where "The max­i­mum rep­re­sentable mag­ni­tude with­out this mod­i­fi­ca­tion would be 240". The orig­i­nal rec­om­men­da­tion was "E4M3 for weight and acti­va­tion ten­sors, and E5M2 for gra­di­ent ten­sors" [117].

So each FP8 tensor car­ries a higher-pre­ci­sion scale that maps its largest value near the format's max­i­mum [117]. The ques­tion is how many values share one. Hopper's trans­former engine scales per tensor [104], Llama 3's FP8 infer­ence per row [6], and DeepSeek-V3 per 1 × 128 acti­va­tion tile and 128 × 128 weight block. With scales that fine, range stops bind­ing and pre­ci­sion wins: DeepSeek-V3 writes "we adopt the E4M3 format on all ten­sors for higher pre­ci­sion", and NVIDIA's MXFP8 recipe does the same [2, 119]. FP8 train­ing recipes (scal­ing gran­u­lar­ity, accu­mu­la­tion pre­ci­sion) are in Sec­tion 5.6.

Microscal­ing builds the block scale into the format. An MX block is 32 ele­ments shar­ing one 8-bit E8M0 scale, a bare power of two [116]. A block format costs

so MXFP8 costs 8.25 bits and MXFP4 4.25 (com­puted). NVFP4 uses 16-ele­ment blocks with an E4M3 scale plus an FP32 scale per tensor, 4.5 bits per value [120], while MXFP4 "can poten­tially lose up to one binade of dynamic range" to power-of-two scales [121]. Black­well's tensor cores read these block scales directly and run FP4 two times faster than FP8 on GB200 and three times on GB300 [121]. Which FP4 vari­ant trains best is open as of Novem­ber 2026: one study finds NVFP4 "pro­vides opti­mal results" [122], others use MXFP4 [123, 124], and AMD's MI355X adver­tises MXFP4 [114].

Narrow inputs need wide accu­mu­la­tors. DeepSeek found that FP8 mat­muls on H800 keep "around 14 bits" of accu­mu­la­tion pre­ci­sion, so its ker­nels pro­mote par­tial sums to FP32 every 128 ele­ments, and it asked for tensor cores that apply group scales them­selves [2], as Black­well's now do.

In 2026 FP32 holds master weights, BF16 most train­ing, FP8 some fron­tier mat­muls and infer­ence, and NVFP4 has reached pre­train­ing: a 12B model on 10T tokens scored 62.58% on MMLU-Pro (a harder ten-option ver­sion of the MMLU knowl­edge exam [125], Sec­tion 10.1) against 62.62% for FP8 [121]. Sec­tion 13.8 sizes Llama 3 405B's weights by format and covers FP8 serv­ing. How train­ing and serv­ing pre­ci­sion enter the scal­ing law is in Sec­tion 6.6.

Fail­ures are numer­i­cal: under­flow under per-tensor scales, sat­u­ra­tion hiding over­flows, and power-of-two scales rounded down that clip block maxima, which the MXFP8 recipe avoids by round­ing up [119]. Out­liers can arrive late: FP8 train­ing of Llama 2 7B diverged "after pro­cess­ing 220B tokens" [126].

2.4 Kernels: fusion, FlashAttention and the per-generation rewrite

Most of a trans­former's FLOPs are in matrix mul­ti­plies, so why does so much engi­neer­ing go into the other oper­a­tions, and into atten­tion? In BERT train­ing, "While tensor con­trac­tions account for over 99% of the arith­metic oper­a­tions per­formed, they con­sti­tute only 61% of the run­time" [127]. Nor­mal­iza­tions, acti­va­tions and soft­max took the rest.

The cure is fusion: if sev­eral oper­a­tions apply to one input, "the input can be loaded once from HBM, instead of mul­ti­ple times for each oper­a­tion" [63]. Horace He calls it "the most impor­tant opti­miza­tion in deep learn­ing com­pil­ers": unfused, x.cos().cos() needs "4 global reads and writes" [77], and fused it needs two. Atten­tion is the extreme case, because stan­dard imple­men­ta­tions "mate­ri­al­ize the matri­ces $𝐒$ and $𝐏$ to HBM" [63], the scores $𝐒=Q{K}^{\top }$ and their soft­max $𝐏$, at $O(s^2)$ memory.

FlashAttention keeps tiles of $Q$, $K$ and $V$ in shared memory and never writes $𝐒$ or $𝐏$ to HBM [63] (its $N$ and $d$ are our $s$ and $d_h$). Soft­max needs a whole row, which one tile does not see, so it uses the online soft­max [128]: each block keeps its row max­i­mum ${\mu }_i$ and sum of expo­nen­tials ${\ell }_i$, and two blocks com­bine exactly,

rescal­ing par­tial out­puts by the same fac­tors. The back­ward pass recom­putes $𝐒$ and $𝐏$ tile by tile, trad­ing cheap FLOPs for HBM traf­fic. The­o­rem 2 counts HBM accesses with $M$ ele­ments of on-chip memory [63]:

With 228 KiB of BF16 shared memory and $d_h=128$, $M/d_h^2\approx 7.1$ (com­puted), the same order as the up to 9 times fewer accesses the paper mea­sured. It made GPT-2 train­ing up to 3 times faster [63].

Each GPU gen­er­a­tion then needed a new kernel. FlashAttention reached "only 25⁠–⁠40% of the the­o­ret­i­cal max­i­mum FLOPs/s" of an A100. FlashAttention-2 cut non-matmul work and par­al­lelized over the sequence, reach­ing 230 TFLOP/s, 73% of peak, in the for­ward pass [38]. On an H100 the same kernel man­aged 35%, against 80⁠–⁠90% for opti­mized mat­muls [82], because Hopper reaches full speed only through its asyn­chro­nous warp­group matrix instruc­tion (WGMMA) [81]. FlashAttention-3 adopted WGMMA, TMA and warp spe­cial­iza­tion, then went after the expo­nen­tial. Per score, two mat­muls cost $4d_h$ tensor-core FLOPs and soft­max one MUFU expo­nen­tial, so

At $d_h=128$ on an H100 that is 4,096/(512 × 16) = 0.5 (com­puted): expo­nen­tials take half the matmul time, as the paper states. Over­lap­ping them with mat­muls took the for­ward pass "from 570 to 661 TFLOPs". For FP8 it rotates $Q$ and $K$ with a random Hada­mard matrix to spread out­liers. The kernel reached 740 TFLOP/s in FP16, 75% of the dense FP16 peak (the same as BF16's), and close to 1.2 PFLOP/s in FP8 [82], about 61% of the FP8 peak (com­puted).

Black­well broke the bal­ance again: B200 dou­bled the matmul rate but kept the MUFU at "16 ops / clock / SM, the same as Hopper", so the ratio rises to 1.0 (com­puted) and expo­nen­tials can no longer hide. FlashAttention-4 com­putes some expo­nen­tials with poly­no­mi­als on the FMA units, skips need­less rescal­ing and uses tensor memory, reach­ing 1,613 TFLOP/s, 71% of a B200, with a deter­min­is­tic mode for rein­force­ment learn­ing [110]. B300 dou­bles the MUFU rate [110], which restores 0.5 if its per-SM matmul rate matches B200's (com­puted, assumed rate).

Figure 10 plots these results. Note FlashAttention-2's drop on the H100, and which chips' expo­nen­tials reach matmul time.

Table 2.4 maps the tools. Small mat­muls cannot fill a chip: on an MI300X "a prob­lem size of approx­i­mately 200 GFLOPs is required to reach peak through­put" [130], while a GPT-2 small MLP matmul at 16K tokens is 77 GFLOPs (com­puted). At that scale launch over­head also shows, so fusion, larger micro-batches and CUDA graphs pay off.

To do it your­self, pro­file first, then use vendor libraries for mat­muls with shapes that tile well, FlashAttention or FlexAttention for atten­tion, torch.com­pile for the rest, CUDA graphs for small steps and Triton for custom fused oper­a­tions. Watch for graph breaks that silently dis­able com­pi­la­tion, non­de­ter­min­is­tic atom­ics, Hopper ker­nels ported to Black­well unchanged, and swiz­zled lay­outs, one of whose dia­grams "cost us three weeks of life that we will not get back" [105].

2.5 Inside the node: NVLink, NVSwitch and PCIe

Why do eight GPUs in one server behave almost like one large GPU for com­mu­ni­ca­tion, and why does the edge of that group matter so much? Inside the server every GPU reaches every other over a switched fabric about nine times faster than the net­work card that leads out­side, so the chat­ti­est traf­fic must stay inside.

Units first

NVLink is full duplex. Each of an H100's 18 links car­ries "25 GB/sec effec­tive band­width in each direc­tion", which NVIDIA sums to "900 GB/sec total band­width" [104], so the GPU sends 450 GB/s while it receives 450 (com­puted). PCIe Gen5 x16 is 128 GB/s in total, 64 each way [104]. Net­work cards are quoted in bits: 400 Gb/s is 50 GB/s [137]. This guide uses one unit, $W$: GB/s, per direc­tion, per accel­er­a­tor.

Sec­tion 11.1 times real col­lec­tives on each rung with Figure 58.

NVSwitch and the NVLink domain

A DGX H100 joins its eight GPUs through four NVSwitches [137] of 64 NVLink ports each [104]. They offer $4\times 64\times 25=\mathrm{6,400}$ GB/s per direc­tion against the $8\times 450=\mathrm{3,600}$ the GPUs can inject, so the switches never block (com­puted). Per-GPU NVLink totals roughly double each gen­er­a­tion: 600 GB/s on A100, 900 on H100 [104], 1,800 on Black­well [139] and a pre­lim­i­nary 3,600 for Rubin [140] (bidi­rec­tional, as of Novem­ber 2026). A GB200 NVL72 rack puts 72 GPUs on one fabric, whose quoted 130 TB/s is $72\times 1.8$ TB/s summed both ways [139] (com­puted). "The term node is over­loaded" [138], so this guide says NVLink domain (or scale-up domain) for the accel­er­a­tors fully con­nected by the fast fabric: 8 on an HGX board (NVIDIA's eight-GPU base­board), 72 on an NVL72 rack, a whole pod on a TPU.

NICs and drives hang off PCIe, and one x16 slot feeds one 400 Gb/s NIC, so each GPU gets its own. Meta writes "There is a 1:1 map­ping between GPUs and NICs" [141]. GPUDirect RDMA lets the NIC read GPU memory directly, but the two "must share the same upstream PCI Express root com­plex" [142]. Wired wrong, traf­fic detours through the CPU and band­width drops with­out an error.

What you actu­ally get

On eight H100s an all-reduce reaches about 370 GB/s only with mes­sages near 10 GB per GPU, and a 58 MB array gets about 150 GB/s [138] (mea­sured). Bench­marks report bus band­width, rescaled by $2(p-1)/p$ so that it com­pares with the link speed [143]: com­pare it with 450, never with 900. The cost of ring and tree col­lec­tives and bus band­width are derived in Sec­tion 11.1. The band­width term for $X$ bytes over $p$ ranks is

That nine­fold gap keeps tensor par­al­lel­ism inside the domain [41, 6]. Sec­tion 11.3 works out TP's com­mu­ni­ca­tion share at TP 8 over NVLink at H100 peak: 0.42 for a stan­dard layer and 0.36 for Llama 3 70B's layer (com­puted).

When NVLink is cut. H800 nodes cut NVLink from 900 GB/s to 400 GB/s [1]. DeepSeek counts that as 200 GB/s (per direc­tion, our read­ing), about 160 achiev­able, against 50 GB/s per InfiniBand NIC, about 4:1 [1]. The V3 report's 3.2 divides the achiev­able 160 by the nom­i­nal 50 [2]. DeepSeek-V3 dropped tensor par­al­lel­ism for 16-way pipe­line and 64-way expert par­al­lel­ism [2]. NVLink faults "often man­i­fest as stalled load/store oper­a­tions within CUDA ker­nels with­out return­ing a clear error code" [6].

2.6 Across the cluster: fabrics, traffic and collectives

How do thou­sands of GPUs get close to full NIC band­width to one another, and why does that take more than enough cables? Give every GPU its own NIC, wire NIC number $i$ of every server to the same leaf switches (a rail), and stack switches in a fat tree with as much band­width going up as coming in. The hard part is load bal­anc­ing, because train­ing traf­fic is the worst case for how switches spread flows.

InfiniBand and RoCE

Both carry RDMA (remote direct memory access), NIC-driven memory-to-memory trans­fers. RoCE v2 car­ries it in UDP/IP over Eth­er­net [141], made loss­less like InfiniBand with pri­or­ity flow con­trol [144]. InfiniBand is faster (2.8 µs against 3.6 µs for 64 bytes within a leaf) but its switches have 64 ports to RoCE's 128 [1]. Meta built a 24,576-GPU clus­ter of each [145], trained Llama 3 405B on RoCE, and says "we tune both of them to pro­vide equiv­a­lent per­for­mance" [6]. Dense train­ing is band­width-bound and hides the latency gap that expert all-to-all exposes.

Fat trees, planes and over­sub­scrip­tion

With $k$-port switches, a non-block­ing tree of two tiers has $k$ leaves of $k/2$ hosts, and one of three tiers reaches $k^3/4$ [146]. In hosts $H$:

For $k=64$ that is 2,048 end­points from 96 switches and 65,536 from 5,120 (com­puted), exactly the fat-tree col­umns of DeepSeek's com­par­i­son [1]. Inde­pen­dent planes avoid a third tier: DeepSeek's eight two-tier planes reach 16,384 GPUs at 4.39 thou­sand dol­lars per end­point against 7.5 thou­sand for three tiers [1], and eight 100 Gb/s planes per 800 Gb/s NIC let 512-port switches reach 131,072 GPUs in two tiers [147].

At $\rho :1$ the upper layer car­ries $1/\rho$ of what the layer below injects, so if every GPU sends across at once each gets

Llama 3 joins eight 3,072-GPU pods, each at full bisec­tion, through a 1:7 layer [6]: 7.1 GB/s per GPU in the worst case (com­puted). Meta's 100K+ fabric moved to 1:2.8 [148], and Alibaba runs 15:1 to its core with only pipe­line traf­fic cross­ing pods [149]. The bet holds because place­ment min­i­mizes cross-pod traf­fic and col­lec­tives stay at hun­dreds of GPUs [141]. Sec­tion 11.7 works out FSDP traf­fic across Llama 3's 1:7 layer: an FP32 reduce-scat­ter of 18.9 GB per GPU, 0.38 s at 50 GB/s, about 2.7 s if all of it crossed, 6⁠–⁠43% of a 6.2 s step.

Why AI traf­fic defeats ECMP

Train­ing traf­fic is "very few, but peri­od­i­cally bursty flows" [149]. Switches spread load with ECMP (equal-cost multi-path), hash­ing each flow onto one uplink, and with few flows that is a lot­tery. Let $f$ equal flows hash onto $u$ uplinks, with $n_{max}$ flows on the busiest. A syn­chro­nous col­lec­tive (Sec­tion 11.1) fin­ishes when that link does:

When $f\ge u$ the through­put is just one over the max-mean ratio, the busiest link's load over the aver­age.

With 16 flows on 16 uplinks, $16(15/16{)}^{16}\approx 5.7$ links sit idle and the busiest car­ries about 3.1 flows, so the col­lec­tive runs at about a third of line rate (com­puted, Monte Carlo). At 1,000 flows the busiest link car­ries about 1.23 times the mean, con­sis­tent with Meta's "aver­age MMR over 1.2 for 1,000 flows on 16 links" [141]. In Figure 11, raise the flows from 16 to 256 and watch the idle grey col­umns fill and the red busiest column fall to about 1.5 times the even share (about 68% of line rate), then try packet spray­ing.

Llama 3's library "cre­ates 16 net­work flows between two GPUs, instead of just one" [6] and hashes on the queue-pair (QP) number, which names an RDMA con­nec­tion, for up to 40% faster AllReduce [141], though 256 ran­domly hashed flows still reach only about 68% of line rate (com­puted). Alibaba fixes each flow's path inside a pod [149]. MRC (Mul­ti­path Reli­able Con­nec­tion, a trans­port) sprays pack­ets over many paths, "elim­i­nat­ing the issue of flow col­li­sions" [147]. Con­ges­tion con­trol is con­tested: Meta runs RoCE with­out DCQCN (ECN-based con­ges­tion con­trol) [141], ByteDance mixes Swift (delay-based) and DCQCN [47], and DeepSeek wants round-trip-time-based or pro­gram­mable con­trol [1].

TPU pods: a torus instead of switches

Each TPU v4 chip has six 50 GB/s links in a 3D torus, and 48 opti­cal cir­cuit switches join 64 cubes of 4 × 4 × 4 chips into 4,096 [112]. The optics buy avail­abil­ity: "With­out OCSes, host avail­abil­ity must be 99.9% to offer rea­son­able slice good­put", and they cost under 5% of the system [112]. A v5p pod has 8,960 chips, and per chip and direc­tion its scale-up beats scale-out by 96×: 1,200 GB/s of bidi­rec­tional ICI (inter-chip inter­con­nect) is 600 each way, against 50 Gb/s (6.25 GB/s) of data-center net­work [88], com­pared with 9× on an H100 (com­puted).

Map­ping par­al­lel­ism onto the ladder

Put the par­al­lel­ism that talks most often on the fastest rung. Llama 3 orders its groups [TP, CP, PP, DP] and keeps FSDP out­er­most because it "can tol­er­ate longer net­work latency" [6]. Mix­ture-of-experts breaks the pat­tern: all-to-all across two nodes is more than four times slower [138], so DeepSeek-V3 sends each token to at most 4 nodes [2], a limit Sec­tion 11.6 derives.

2.7 Clusters at scale: power, storage and failures

What does a train­ing clus­ter draw from the grid, what must its stor­age absorb, and how often does a syn­chro­nous job on it break? About twice its chips' power times the build­ing's over­head, check­point bursts, and every few hours. Meta built two 24,576-GPU clus­ters [145], of which Llama 3 405B used up to 16K [6], and xAI says Colos­sus reached 200,000 GPUs [150] (com­pany claim). A clus­ter is not a job.

Pilz et al. esti­mate peak facil­ity power from chip TDP [153]:

Here $\kappa =2.03$ covers CPUs, switches and stor­age (cal­i­brated on DGX H100). PUE (power usage effec­tive­ness), facil­ity over IT power, was 1.14 for AI data cen­ters in 2023 [153], 1.09 for Google's 2025 fleet and 1.54 in a survey [154]. Llama 3's 16,384 GPUs at 700 W [6] and Pilz et al.'s 2024 PUE of 1.12 [153] give $\mathrm{16,384}\times 0.7\times 2.03\times 1.12\approx 26$ MW, of which the GPUs draw 11.5 MW (com­puted). Colos­sus at PUE 1.10 gives 313 MW against the roughly 300 MW Pilz et al. report (com­puted). The mul­ti­plier is debat­able: the DGX H100 name­plate, 10.2 kW for eight 700 W GPUs, implies 1.82 (com­puted), so treat 2.03 as an upper esti­mate for air-cooled servers and use rack name­plates for rack-scale sys­tems. The clus­ter panel of Figure 12 applies either mode to any chip.

At 120 kW a rack, air runs out, and NVL72 trays are liquid-cooled through cold plates [151]. Llama 3 405B's through­put varied by 1⁠–⁠2% over the day, because higher mid-day tem­per­a­tures affected the GPUs' dynamic volt­age and fre­quency scal­ing [6]. Thou­sands of GPUs com­put­ing and then wait­ing together swing the load. For Llama 3 that "can result in instant fluc­tu­a­tions of power con­sump­tion across the data center on the order of tens of megawatts, stretch­ing the limits of the power grid" [6]. The swings sit at 0.2⁠–⁠3 Hz, near grid res­o­nances, and smooth­ing costs energy: hold­ing GB200 power at a floor of 90% of TDP added 10.5% on one trace [155].

Llama 3's stor­age holds 240 PB and sus­tains 2 TB/s, with 7 TB/s peaks [6]. That is not for the text: at 16,777,216 tokens per 6.2 s step the job reads about 2.7M tokens per second, about 11 MB/s of 4-byte token IDs (com­puted). It is for "highly bursty check­point writes that sat­u­rate the stor­age fabric for short dura­tions" [6]. Check­point size, write band­width and the Young inter­val are in Sec­tion 12.3, and data-loader through­put is in Sec­tion 12.9. Fail­ures grow with node count, and Llama 3 saw one unex­pected inter­rup­tion every 3.1 hours [6] (com­puted, 419 in 54 days). Fail­ure rates, the 1/N law and Llama 3's 466 inter­rup­tions are in Sec­tion 12.1.

2.8 The accelerator landscape and how to read a spec sheet

How do you com­pare accel­er­a­tors from dif­fer­ent ven­dors with­out being fooled, and which num­bers decide the choice for train­ing? Nor­mal­ize first, then com­pare ratios. The roofline and the ridge point are derived in Sec­tion 1.7. Apply them here with­out re-deriv­ing.

• Dense, not sparse. H100's BF16 1,979 car­ries "With spar­sity" [4].
• Same format. TPU v5p and v6e list FP8 equal to BF16 [156], because FP8 was "emu­lated for v4 and v5p, but natively sup­ported for Iron­wood" [152].
• Per chip, not per board or rack. HGX B200's 36 sparse PFLOP/s for eight GPUs is 2.25 dense each [85].
• Peak is units × FLOP per clock × clock. MI355X: $256\times \mathrm{4,096}\times 2.4$ GHz $=\mathrm{2,516.6}$ TFLOP/s [114] (com­puted).
• Per direc­tion, and GB or GiB. NVLink's 900 GB/s is both ways, and Iron­wood's 192 GiB is 206 GB (com­puted).
• Ship­ping, not announced. Rubin's 4 PFLOP/s is an announce­ment [85].
• Same arith­metic. Iron­wood's "for a total of 42.5 Exaflops" is FP8, set beside El Cap­i­tan's FP64 [152].
• TDP is not aver­age draw. Idle power is about 20% of TDP on A100, H100 and MI300, 80% on Cere­bras [157].
• Date it. The table was read in Novem­ber 2026.

Three ratios carry most of the table (all com­puted). The ridge point $I_{\text{chip}}^{*}=F_{\text{dense BF16}}/W_{\text{HBM}}$ puts 10 of the 14 chips between 200 and 320 FLOPs per byte. TPU v5p, Rubin, Gaudi 3 and TPU v6e are the out­liers. A large train­ing matmul clears every ridge (Llama 3 8B's MLP over 8,192 tokens does about 2,294 FLOPs per HBM byte), while decode sits far below, which is why H200 and MI325X add band­width with­out com­pute (Sec­tion 13.2). Com­pute per watt rose from 1,414 TFLOP/s per kW on H100 to 2,250 on B200. Memory per FLOP sep­a­rates ven­dors: 81 GB per dense PFLOP/s on H100 against 147 on MI300X, so a model shards over fewer AMD chips. In Figure 12, find the diag­o­nal ridge lines, then switch to FP8: hollow mark­ers (no FP8 speedup) stay put.

Four design fam­i­lies share the market. Eight-chip boards with a NIC per chip: HGX, MI300X and MI355X, and Gaudi 3, whose Eth­er­net ports sit on the chip [164]. Rack-scale domains: NVL72 runs tensor and expert par­al­lel­ism 72-way on NVLink [139], with one 400 Gb/s NIC per GPU out­side, an 18:1 gap [151] (com­puted). Torus-con­nected chips: TPU pods, and Trainium2 with 64 chips per UltraServer at 1.28 TB/s of NeuronLink each [162]. SRAM-cen­tric designs such as Cere­bras' 44 GB wafer [165] trade capac­ity for band­width. Ven­dors lead with FLOP/s, but DeepSeek asks for scale-up and scale-out in "a uni­fied frame­work" [1], and a 2026 com­par­i­son finds the best plat­form shifts with batch and model size [157]. For train­ing, read band­width, memory and the scale-up domain before peak FLOP/s.

An H100 spec page lists "1,979 teraFLOPS" for BF16. You mea­sure 600 TFLOP/s on a large matmul. What share of peak is that?

About 61%, not 30%. The 1,979 assumes 2:4 spar­sity ("With spar­sity"), which dense train­ing does not use [4]. The dense peak, 989.5, matches $132\times \mathrm{4,096}\times 1.83$ GHz. The sparse figure halves every MFU.

Why did early FP8 recipes put gra­di­ents in E5M2 (max­i­mum 57,344), while DeepSeek-V3 uses E4M3 (max­i­mum 448) every­where?

It depends on how many values share a scale. With one scale per tensor, gra­di­ents span many binades, so E5M2's 32 binades win [117]. With one scale per 1 × 128 tile or 128 × 128 block, each group fits E4M3's range and its extra man­tissa bit wins [2].

FlashAttention recom­putes the score matrix in the back­ward pass, so it does more FLOPs. Why is it faster?

Stan­dard atten­tion is bound by HBM, not FLOPs. At 8,192 tokens one head's BF16 scores are 134 MB, and moving them costs about 4.6 times the matmul time on an H100 (com­puted). Tiling in shared memory cuts HBM accesses from $\Theta (s{d}_h+s^2)$ to $\Theta (s^2{d}_h^2/M)$ [63], about 7 times fewer on an H100.

A vendor page lists NVLink at "900 GB/s" and the NIC at "400G". How much faster is NVLink for one H100 send­ing data?

Nine times, not 18. 900 GB/s counts both direc­tions, so 450 each way [104], and 400 Gb/s is 50 GB/s [137]. On an H800, whose NVLink totals 400 GB/s, the like-for-like ratio is 4 [1].

Llama 3's stor­age sus­tains 2 TB/s. Is that band­width for the train­ing data?

No. The run read about 11 MB/s of token IDs, roughly 185,000 times less (com­puted). The band­width serves check­point bursts: a 405B check­point is about 5.7 TB at 14 bytes per param­e­ter, 6.5 TB at 16 (Sec­tion 12.3), writ­ten by every GPU at once [6].

3. Data: from the raw web to training tokens

The train­ing dis­tri­bu­tion is built, one fil­ter­ing and sam­pling deci­sion at a time. Follow text from the web crawl to token ids, then test what clean­ing, dedu­pli­ca­tion, tok­eniza­tion and mix­ture weights do to the model you can afford to train.

Llama 3 was pre­trained on about 15 tril­lion tokens, and every lab throws away most of the web it down­loads. Llama 3 used "about 15T mul­ti­lin­gual tokens, com­pared to 1.8T tokens for Llama 2", in a mix of roughly 50% gen­eral knowl­edge, 25% math and rea­son­ing, 17% code and 8% mul­ti­lin­gual text [6]. Sec­tion 3.5 returns to the mix. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

The raw mate­rial is plen­ti­ful. Common Crawl, a non­profit web archive, releases a crawl of 200 to 400 TiB of text every one or two months, and the April 2024 crawl held 2.7 bil­lion pages [166]. The usable mate­rial is scarce. FineWeb, built from 96 snap­shots and counted in GPT-2 tokens, had about 36T tokens after base fil­ter­ing, 20T after dedu­pli­cat­ing each snap­shot and 15T after more heuris­tic fil­ters [5]. Its edu­ca­tional sub­sets keep 5.4T and 1.3T [166], so FineWeb-Edu is $1.3/36=3.6\%$ of the base-fil­tered tokens (com­puted). The step from raw HTML to 36T is not reported. DCLM's pool of 200B doc­u­ments and 240T tokens shrinks to 19.9% of doc­u­ments after heuris­tics, 13.7% after dedu­pli­ca­tion and 1.4% after a qual­ity clas­si­fier [167]. RefinedWeb keeps 11.67% of its Common Crawl doc­u­ments [168].

Figure 13 draws the three fun­nels stage by stage. Switch pre­sets and find the big­gest cut: lan­guage iden­ti­fi­ca­tion in RefinedWeb, dedu­pli­ca­tion in FineWeb, the clas­si­fier in DCLM.

Every stage is a mod­el­ing deci­sion, judged only by train­ing a model on the result, so data work is an exper­i­men­tal sci­ence run with small proxy models. On long runs the bind­ing con­straint becomes unique tokens: how much you can repeat, syn­the­size and legally use.

Besides Llama 3, the chap­ter fol­lows OLMo 2, whose 3.9T-token mix is over 95% web [7], and GPT-2 124M trained on FineWeb-Edu's 10B-token sample [169]. Sec­tions 3.1 to 3.3 clean the text, 3.4 reads the open datasets, 3.5 and 3.6 mix and repeat it, 3.7 and 3.8 tok­enize it, and 3.9 tests data deci­sions.

3.1 From Common Crawl to clean text

What exactly do you down­load from Common Crawl, and why does the step that turns HTML into text already change the model you get? Each crawl ships in three forms. WARC files hold the raw HTTP responses with full HTML, and WET files hold plain text that Common Crawl extracted itself [5]. WAT files hold com­puted meta­data such as head­ers and links [170].

WET is the cheap option, and it is worse. FineWeb found that WET "retained too much boil­er­plate and menu text" [5], where boil­er­plate means the menus, foot­ers, cookie notices and ads around a page's main con­tent. On one snap­shot WET gave about 254B tokens against about 200B from WARC with the trafi­latura extrac­tor: 25% more text and a much worse model [166]. DCLM mea­sured the gap at 1B-1x with iden­ti­cal down­stream fil­ters: WET 20.7 Core, resili­parse 24.1, trafi­latura 24.5, with resili­parse about eight times faster [167]. The first preset of Figure 14 plots these runs.

Extrac­tors dis­agree

An extrac­tor decides which HTML blocks are main con­tent. Common choices are trafi­latura [171], resili­parse [172] and jusText, which labels each block as boil­er­plate or con­tent [173]. Llama 3 wrote its own parser, kept image alt text because math is often ren­dered as images, and stripped mark­down because it hurt models trained mainly on web text [6].

Pre­ci­sion is not the only goal. On 13 snap­shots, Nemotron-CC found that fil­tered jusText kept 104B high-qual­ity tokens against 80B for fil­tered trafi­latura, and unfil­tered jusText kept 127B [174]. The extrac­tor that looks dirt­ier can carry more of the text you want. A 2026 study ran the DCLM-Base­line pipe­line on three extrac­tors and found that 61% of sur­viv­ing pages were kept by just one of them. Their union raised token yield by up to 71% at equal Core scores. On table-heavy pages the extrac­tor moved WikiTableQuestions by up to 10.3 points, and on code pages HumanEval by up to 3.6 [173].

Lan­guage iden­ti­fi­ca­tion and URL fil­ters

Lan­guage iden­ti­fi­ca­tion (LID) pre­dicts each doc­u­ment's lan­guage. Most pipe­lines use fastText [175, 176], a linear clas­si­fier over char­ac­ter n-grams, and keep a doc­u­ment when the top lan­guage is the target and its prob­a­bil­ity clears a thresh­old.

LID is RefinedWeb's big­gest single cut: it removes about half of the doc­u­ments that reach it. It dou­bles as a junk filter, because a score below 0.65 "usu­ally cor­re­sponds to pages with­out any nat­u­ral text" [168]. It fails worst for low-resource lan­guages, whose cor­pora can be more than 90% false pos­i­tives from a related high-resource lan­guage [179]. URL fil­ters run first because they need only the address. RefinedWeb's URL filter, a 4.6M-domain block­list plus a score for flagged words in the URL, removes just 2.24% of doc­u­ments [168], and Llama 3 drops domains likely to hold unsafe or adult con­tent or a lot of per­son­ally iden­ti­fi­able infor­ma­tion (PII) [6].

The default: extract from WARC with resili­parse or trafi­latura, check tables, code and math sep­a­rately, and take a union of extrac­tors if data is short. FineWeb's blog calls WET a fair trade-off for small bud­gets, since extrac­tion is among the costli­est steps [166]. Extrac­tors that drop <pre>, <table> or MathML silently delete code and math. None removes machine-gen­er­ated pages: in C4, "the single-most rep­re­sented web­site in the corpus is patents.google.com" [181].

3.2 Quality filtering: rules, classifiers and their side effects

After extrac­tion and lan­guage iden­ti­fi­ca­tion, most of the text is still not some­thing a model should imi­tate. How do you sep­a­rate good from bad, and what does qual­ity mean? Rules and models both answer it, and one test judges both: train on what they keep.

Heuris­tic rules

A heuris­tic filter is a thresh­old on a simple doc­u­ment statis­tic. C4 kept only lines ending in ter­mi­nal punc­tu­a­tion, pages of at least 3 sen­tences and lines of at least 5 words, and dropped pages with a listed bad word, place­holder text or a curly bracket [177]. Gopher keeps doc­u­ments of 50 to 100,000 words with a mean word length of 3 to 10 char­ac­ters, caps the symbol-to-word ratio at 0.1 and limits repeated lines [182].

Thresh­olds chosen by eye are guesses, so FineWeb mea­sured them [5]. It took a high- and a low-qual­ity ver­sion of one snap­shot (Sec­tion 3.3 explains where the low one came from), com­puted over 50 sta­tis­tics per doc­u­ment, and set thresh­olds where the low-qual­ity his­togram dom­i­nates, then kept those that won 28B-token abla­tions. The three win­ners target lines with­out final punc­tu­a­tion, dupli­cated lines and very short lines. Together they removed about 22% of tokens and raised the aggre­gate score by about 1%, while C4's punc­tu­a­tion rule alone cost about 30% of tokens [5]. Rules also delete good text: Nemotron-CC's heuris­tic and per­plex­ity fil­ters removed 18.1% of the tokens the FineWeb-Edu clas­si­fier rates high qual­ity [174].

Model-based fil­ters

A model-based filter scores each doc­u­ment against a ref­er­ence set, the text it learns to treat as good. GPT-3 trained a logis­tic-regres­sion clas­si­fier to sep­a­rate curated text from raw Common Crawl, then kept each doc­u­ment if np.random.pareto(9) exceeded one minus its score $q$, so as to "still include some doc­u­ments that were out of dis­tri­bu­tion" [36]. NumPy's pareto(9) draws a Lomax vari­able $X$ with $P(X>x)=(1+x{)}^{-9}$, so (derived, and con­firmed by sim­u­la­tion)

FineWeb-Edu dis­tills an LLM's judg­ment into a cheap scorer [5]. A small regres­sion head, trained on 410,000 edu­ca­tional-value rat­ings (0 to 5) from Llama-3⁠–⁠70B-Instruct, reached an F1 of 82% at thresh­old 3. Scor­ing all 15T FineWeb tokens took 6,000 H100-hours, 0.4% of the 1.46M H100-hours reported for train­ing Llama 3.1 8B (com­puted) [11]. At 1.71B param­e­ters and 350B tokens, MMLU rose from 33% to 37% and ARC from 46% to 57%, and the price was a skew toward edu­ca­tion and his­tory topics [5].

DCLM com­pared filter fam­i­lies at 1B-1x from a 27.5-Core base­line. PageRank scored 26.1, AskLLM (prompt­ing a model to judge each doc­u­ment) 28.6, per­plex­ity fil­ter­ing (keep­ing doc­u­ments a small ref­er­ence lan­guage model finds likely) 29.0, and a fastText clas­si­fier whose pos­i­tives were OpenHermes 2.5 instruc­tion data and answers from the ELI5 forum 30.2 [167]. The ref­er­ence set mat­tered more than the clas­si­fier. At 7B-1x those pos­i­tives scored 41.0 Core against 37.5 for a GPT-3-style mix, 35.7 for Wikipedia and 34.7 for OpenWebText2, and keep­ing the top 10% beat the top 15% and 20% [167]. Why is open. One hypoth­e­sis is that their ques­tion-and-answer format resem­bles the bench­marks.

Figure 14 puts these results side by side. Look at the sizes: most fil­ters move Core by -1.4 to +2.7 points, the pos­i­tives by up to 6.3, and each mark is a single run with­out error bars.

Strict fil­ters also keep too little for long runs. Nemotron-CC notes that the FineWeb-Edu and DCLM clas­si­fiers label only 8% and 11% of tokens high qual­ity, so it ensem­bles three clas­si­fiers into 20 score buck­ets and weights the buck­ets in the mix instead of dis­card­ing them [174] (Sec­tion 3.4). It left FineWeb-Edu's clas­si­fier out because Llama 3 anno­ta­tions trained it [174]: the license of the model that wrote the labels can con­strain a clas­si­fier trained on them.

Side effects

Fil­ters remove more than junk. C4's bad-words list removed African Amer­i­can Eng­lish and His­panic-aligned Eng­lish at 42% and 32%, against 6.2% for White-aligned Eng­lish [181], which is why Gopher used SafeSearch instead of word lists [182]. Tox­i­c­ity fil­ter­ing raises loss most for marginal­ized groups [183], and qual­ity fil­ter­ing "sub­stan­tially increases both toxic gen­er­a­tion and down­stream per­for­mance" [184].

As of Novem­ber 2026 a sound default is Gopher-style rules with thresh­olds set by con­trast, then a fastText or small-encoder clas­si­fier trained on LLM labels or instruc­tion-style pos­i­tives. Keep the top 10⁠–⁠20% for short runs, weight score buck­ets for long ones, and report tokens removed next to every score.

3.3 Deduplication

The web repeats itself. How do you find dupli­cates among bil­lions of doc­u­ments, how aggres­sive should you be, and does more dedu­pli­ca­tion always help? Dupli­cates waste com­pute and cause mem­o­riza­tion. Over 1% of the unprompted output of a 1.5B model trained on C4 was copied ver­ba­tim from train­ing data, and dedu­pli­ca­tion cut that ten­fold [185]. A sequence seen 10 times is gen­er­ated about 1,000 times more often than one seen once [186].

Exact dupli­cates and Bloom fil­ters

Start cheap. In Dolma, exact URL dedu­pli­ca­tion removed 53.2% of doc­u­ments [178]. Llama 3 removes lines that appear more than 6 times in each bucket of 30M doc­u­ments, which strips boil­er­plate [6]. To ask "seen before?" over tril­lions of items, use a Bloom filter: $n_{\text{bits}}$ bits and $n_{\text{hash}}$ hash func­tions. Insert­ing an item sets its $n_{\text{hash}}$ bits, and a query answers "seen" if all are set, so there are no false neg­a­tives. After $n$ inser­tions a bit is still zero with prob­a­bil­ity about $e^{-n_{\text{hash}}n/n_{\text{bits}}}$, so the false-pos­i­tive rate is

The opti­mum leaves half the bits set, which gives the last two for­mu­las (derived). DCLM notes that 1T tokens at $\epsilon ={10}^{-12}$ "would require 6.5TB of RAM" [167]. The for­mula gives 40 hashes and $7.19\times {10}^{12}$ bytes, which is 6.54 TiB, so that TB is binary. At $\epsilon =0.01$ the corpus needs 1.20 TB (com­puted). Its Bloom-filter dedu­pli­ca­tion matched MinHash plus suffix arrays within 0.2 Core [167].

MinHash and band­ing

Near-dupli­cates differ by a date, a name or a menu. Rep­re­sent each doc­u­ment by its shin­gles, the set of its word 5-grams, and mea­sure over­lap with Jac­card sim­i­lar­ity $J(A,B)=|A\cap B|/|A\cup B|$. Com­par­ing all pairs is impos­si­ble, so MinHash com­presses each set into a short sig­na­ture [187]. Apply a random per­mu­ta­tion $\pi$ to all shin­gles and keep each set's min­i­mum. Every ele­ment of $A\cup B$ is equally likely to come first, and the minima agree exactly when that ele­ment is shared:

With $K$ hash func­tions, the share of match­ing minima esti­mates $J$ with stan­dard devi­a­tion $(J(1-J)/K{)}^{1/2}$, 0.046 at $J=0.6$ and $K=112$ (com­puted). Band­ing turns that into a cheap lookup [188]: split the $K=n_b{n}_r$ values into $n_b$ bands of $n_r$ rows, and flag a pair if every row of some band matches. One band matches with prob­a­bil­ity $J^{n_r}$, so

The curve is an S with its thresh­old $J_{\text{thr}}$ near the steep­est point. FineWeb's 112 hashes in 14 bands of 8 flag pairs at $J$ = 0.7, 0.75, 0.8 and 0.85 with prob­a­bil­ity 56%, 77%, 92% and 98.8% [5], exactly what the for­mula gives, with $J_{\text{thr}}=0.72$ (com­puted). Flagged pairs form a graph, and pipe­lines keep one doc­u­ment per con­nected com­po­nent. Nor­mal­iz­ing text first (low­er­case, strip punc­tu­a­tion and accents) raises recall [185, 168]. Lee et al. and RefinedWeb use 9,000 hashes in 450 bands of 20, flag­ging 76% at 0.75 and 99.4% at 0.8 [185, 168]. That sharper cut costs $\mathrm{9,000}/112=80$ times the hash­ing and stor­age (com­puted), and FineWeb judged the sav­ings worth "the higher uncer­tainty on doc­u­ments near the thresh­old" [5].

In Figure 15, move the slid­ers. More rows per band push the thresh­old right and steepen the curve, more bands pull it left, and the read­out shows the cost in hashes.

Exact-sub­string dedu­pli­ca­tion catches shared spans that doc­u­ment-level MinHash misses. It builds a suffix array over the con­cate­nated corpus and removes repeated spans of 50 or more tokens. At 8 bytes per token it needs about 8 TB per tril­lion tokens (com­puted), and it removed 7.18% of C4's tokens [185]. Seman­tic dedu­pli­ca­tion, which drops near neigh­bors in an embed­ding space, scored 27.1 against 27.5 Core on DCLM's web text [167].

Others dedu­pli­cate glob­ally and do fine. Llama 3 runs global MinHash [6], and FineWeb2 dedu­pli­cates each lan­guage glob­ally, then repeats doc­u­ments by clus­ter size with weights up to 10 [179]. Global dedu­pli­ca­tion is safe when qual­ity fil­ter­ing or dupli­cate-aware upsam­pling fol­lows, and the copy count is itself a qual­ity signal.

The default: URL dedu­pli­ca­tion first, line or para­graph hashes for boil­er­plate, MinHash on word 5-grams near $J$ = 0.7⁠–⁠0.8 per snap­shot (or global with upsam­pling), and exact sub­strings at 50 tokens if memory allows.

3.4 What the open datasets taught us

Read in order, what did each open web corpus teach, and how far should you trust the head­line num­bers? The open cor­pora read like a series of exper­i­ments, and Table 3.2 gives one lesson per row. Sizes are in each source's own unit and tok­enizer, so com­pare rows only roughly.

Fil­tered web beats curated mix­tures: RefinedWeb showed that "prop­erly fil­tered and dedu­pli­cated web data alone can lead to pow­er­ful models" [168], and at 7B DCLM mea­sured RefinedWeb at 36.9 on its Core aver­age (Sec­tion 3.2) against 34.2 for C4, 35.0 for Dolma-V1 and 35.3 for RedPajama (reported) [167].

Read­ing the head­line num­bers

DCLM-Base­line trains a 7B model "from scratch to 64% 5-shot accu­racy on MMLU with 2.6T train­ing tokens" (reported), but that model also saw StarCoder code, ProofPile2 math and math-heavy cooldowns, so it is not fil­tered web alone [167]. MMLU is a four-way mul­ti­ple-choice knowl­edge test (Chap­ter 10). DCLM's own head-to-head puts DCLM-Base­line at 50.8 MMLU against 37.3 for FineWeb-Edu (7B, 0.28T tokens) [167]. Nemotron-CC's 8B model scores 70.3 against 65.3 for Llama 3.1 8B, on the authors' own har­ness (reported) [174]. Remov­ing detected MMLU over­laps from DCLM-Base­line did not lower its score (51.8 to 52.7) [167]. So ask of every number: base model, mostly mul­ti­ple-choice Eng­lish bench­mark, whose har­ness? And could an edu­ca­tional filter raise MMLU partly by select­ing text that reads like exam mate­rial? That is an open ques­tion.

The long-hori­zon prob­lem

Strict fil­ters shine at short token hori­zons and can starve a 15T-token run. Nemotron-CC was built for that prob­lem: its 6.3T tokens match DCLM on MMLU with four times as many unique real tokens (reported) [174]. Its authors also state that "Both DCLM and FineWeb-Edu con­tain around 80% near-dupli­cates (1T and 0.2T unique tokens, respec­tively)" [174], a com­pet­ing team's claim that this guide has not checked.

A newer prob­lem: the web is fill­ing with model text, which is why DCLM-Pool omits crawls from 2023 on [167]. As of late 2026, after FineWeb fil­ter­ing, the Pan­gram detec­tor labels 27.5% of June 2026 web tokens as AI-gen­er­ated and 31.1% by August, and such tokens raise loss almost imme­di­ately for models trained on large bud­gets of human text (reported, detec­tor-depen­dent) [194]. Treat crawl date as a filter vari­able.

3.5 Mixtures: how much web, code and math

A pre­train­ing corpus is a set of domains of very dif­fer­ent sizes and qual­i­ties. Given a token budget, what frac­tion of train­ing tokens should come from each? Not a share pro­por­tional to size. GPT-3 chose its weights so that "some datasets are seen up to 3.4 times during train­ing while other datasets are seen less than once" [36]. A mix­ture is a sam­pling dis­tri­bu­tion $P_w={\sum }_i{w}_i\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{f}({𝒟}_i)$ over domain datasets ${𝒟}_i$, with weights $w_i\ge 0$ that sum to 1 (the sim­plex ${\Delta }^k$ over $k$ domains) [195]. Small, dense sources get upsam­pled and big, noisy ones down­sam­pled.

A run of $D$ tokens draws $w_{i}D$ tokens from domain $i$, and one pass (one epoch) over the domain con­sumes its $U_i$ unique tokens, so the epochs over domain $i$ are

assum­ing token shares and no fur­ther fil­ter­ing during train­ing. phi-4 states this rule [196], and it repro­duces phi-4's table: with $D\approx 10$T, syn­thetic data at 40% over 290B unique tokens is $0.4\times 10\text{T}/290\text{B}=13.8$ epochs and web at 15% over 1.3T is 1.15, against the printed 13.8 and 1.2 (com­puted). Check units first: GPT-3's weights count exam­ples, so WebText2's 22% of 300B tokens over 19B unique gives 3.5 epochs where its table says 2.9 (com­puted).

Llama 3 chose its mix with a topic clas­si­fier that down­sam­ples over-rep­re­sented cat­e­gories, small-model scal­ing-law exper­i­ments, and recent web data added late [6]. A pro­duc­tion mix is a sched­ule.

Learn­ing the weights

DoReMi trains a small ref­er­ence model on default weights, then a small proxy with Group DRO (dis­tri­bu­tion­ally robust opti­miza­tion, min­i­miz­ing the worst-case loss over groups) on each domain's excess loss, proxy minus ref­er­ence [195]:

Here ${\ell }_{\theta }(x)$ is the proxy's loss on exam­ple $x$ and $|x|$ its length in tokens. In prac­tice the per-token excess loss is clipped at 0 before aver­ag­ing. Excess loss is high where a domain is learn­able but not yet learned, so noise and already-easy domains both lose weight. Weights from a 280M proxy gave an 8B model 6.5 points more aver­age one-shot accu­racy than The Pile's defaults and reached the base­line's accu­racy 2.6 times faster, for 8% of the large run's com­pute (reported) [195].

Regres­sion meth­ods fit loss as a func­tion of the mix. Data mixing laws fit ${ℒ}_i(w)=c_i+k_i\exp \left({\sum }_j{t}_{ij}{w}_j\right)$ per val­i­da­tion domain $i$, where a neg­a­tive $t_{ij}$ means domain $j$ helps domain $i$, and find a mix the default needs 48% more steps to match [200]. RegMix fits 512 prox­ies of 1M param­e­ters for nearly 2% of one 1B model's FLOPs and finds that web data, not Wikipedia, cor­re­lates best with down­stream scores [201]. Olmix (2026) needs proxy runs linear in the number of domains [202]. But Aioli found that "no exist­ing method con­sis­tently out­per­forms a simple strat­i­fied sam­pling base­line in terms of aver­age test per­plex­ity" (equal weight per domain) [203], and the best mix is scale-free only if the mix­ture enters the scal­ing law addi­tively [204]. Con­firm a learned mix at one larger scale.

"High qual­ity" is rel­a­tive. Mixing Wikipedia, books, StackExchange, arXiv and GitHub raised DCLM's Core score for C4 (+2.2) and RefinedWeb (+1.4), but low­ered it for DCLM-Base­line, from 31.1 to 29.9 [167]. A clas­si­fier that already selects ref­er­ence-like text leaves curated sources little to add.

Code and math

Code gets its own pipe­line. StarCoder2⁠–⁠15B trained on "913B+ unique tokens" after hon­or­ing opt-outs [199]. Fill-in-the-middle (FIM) reorders a doc­u­ment's prefix, middle and suffix with sen­tinel tokens so the model learns to infill. In a sweep of six 200B-token runs, rea­son­ing peaks at 25% code (3.4% above the text-only model), while world knowl­edge falls 3.4% at 25% code and 31% at 75%, all rel­a­tive changes [205]. Mod­er­ate shares help, and high shares dis­place the text that car­ries knowl­edge.

Math is rare on the web, and naive extrac­tion destroys its nota­tion, so OpenWebMath keeps LaTeX [206]. DeepSeekMath's iter­a­tive recall is the method to copy: train a fastText clas­si­fier on 500,000 seed math pages against 500,000 random ones, score the crawl, add pages from domains with many hits to the seeds, retrain and repeat. Four rounds gave 120B tokens [207]. Upsam­pling such data over the last 10⁠–⁠20% of train­ing also pays [208] (Chap­ter 7).

How to do it your­self. Start from a pub­lished mix, com­pute $n_{\text{ep},i}$ for every domain before launch, cap small domains at about four epochs, keep 10⁠–⁠25% code, and make any learned mix beat the nat­u­ral and strat­i­fied base­lines. Watch for shares in mixed units, domains a filter silently exhausted, and mixes tuned on the reported bench­marks.

3.6 When unique data runs out: repetition, synthetic data and consent

When the clean, unique data runs out, how much is a repeated token worth, and can models write the rest? A 15T-token run on FineWeb-Edu alone is 11.5 epochs. Intu­ition says the first repeat is nearly as good as new data and the hun­dredth nearly worth­less, and a simple model of that fits real runs.

The rep­e­ti­tion law

Sup­pose each rep­e­ti­tion of a token is worth a frac­tion $1-\delta$ of the pass before. With $U_D$ unique tokens seen for $n_{\text{ep}}$ epochs, so $R_D=n_{\text{ep}}-1$ rep­e­ti­tions, the effec­tive data $D^{\prime }$ (the fresh tokens that would give the same loss) is a geo­met­ric sum:

Define $R_D^{\star }=(1-\delta )/\delta$. For small $\delta$, $(1-\delta {)}^{R_D}\approx e^{-R_D/R_D^{\star }}$, which gives the form Muen­nighoff et al. fit [209]:

For few rep­e­ti­tions $D^{\prime }\approx D$, and for many it sat­u­rates at $U_D(1+R_D^{\star })$. A fit on 182 runs from 7M to 9B param­e­ters and 1 to 500 epochs gives $R_D^{\star }=15.387756$, so each pass is worth $\delta =1/(1+R_D^{\star })=6.1\%$ less than the last (com­puted). The con­stants come from C4 with the GPT-2 tok­enizer, so do not trans­plant them blindly.

The num­bers to remem­ber (com­puted): 4 epochs are worth 3.73 times the unique data (93% of fresh), 16 epochs 10.58 times (66%), and no number of epochs beats 16.39 times. The 4th pass alone is worth 0.85 of a fresh one, the 16th 0.39, and FineWeb-Edu's 11.5 epochs are worth 8.6 fresh ones. Figure 16 plots the law: look for where the curve leaves the fresh-data diag­o­nal (around 4 epochs) and flat­tens toward the ceil­ing (past 16), then switch the decay con­stant or check a real mix­ture in the plan­ner.

The paper reports neg­li­gi­ble loss changes up to 4 epochs and "Mean­ing­ful gains" up to about 16 [209]. Its 8.7B model trained for four epochs on 44B unique tokens ended with only 0.5% higher loss than one epoch over 178B (reported). The full law, which also decays param­e­ters beyond the com­pute-opti­mal size for the data ($R_N^{\star }=5.309743$), is

and with the paper's C4 con­stants it pre­dicts about 0.7% when both runs see 178B tokens (0.75% if the four-epoch run gets only 4 x 44B = 176B), slightly more than mea­sured (com­puted).

The law has edges, and its fit dropped runs where rep­e­ti­tion hurt [209]. Repeat­ing 0.1% of the data 100 times degraded an 800M model to the level of a 400M one [210], which is what sur­viv­ing near-dupli­cates do (Sec­tion 3.3). Over many epochs larger models over­fit more [211], and loss can rise with epochs unless weight decay is 30 times the stan­dard value [212]. The con­stant also depends on the set­ting: 31.93 for autore­gres­sive and 512.85 for masked dif­fu­sion models in another fit [213].

Syn­thetic data

When rep­e­ti­tion is used up, a model can write more data. A strong gen­er­a­tor can write new doc­u­ments such as text­books, or a model can rephrase real pages, which stay the source of knowl­edge. phi-1, a 1.3B code model trained with syn­thetic text­book-style data, "attains pass@1 accu­racy 50.6% on HumanEval" [214]. phi-4 cre­ated about 400B unweighted syn­thetic tokens (290B unique in its mix table) and gave syn­thetic data 40% of train­ing. At an equal token budget, 12 epochs of syn­thetic data beat 4 epochs plus more unique web tokens [196]. The authors argue syn­thetic text is easier to learn from because "each token gen­er­ated by a lan­guage model is by def­i­ni­tion pre­dicted by the pre­ced­ing tokens, making it easier for a model to follow the result­ing rea­son­ing pat­terns" [196]. For rephras­ing, WRAP trains on real and rewrit­ten C4 "in a 1:1 ratio" for more than 10% better per­plex­ity [215], and Nemotron-CC adds 1.9T rephrased tokens [174].

Syn­thetic tokens cost real com­pute. WRAP gen­er­ates "3M tokens per hour on a single A100", so its 85B tokens take $85\times {10}^9/(3\times {10}^6)$, or 28,333 GPU-hours (com­puted) against the paper's "about 25K" [215]. Gen­er­at­ing a token with a 7B model costs about $2N=1.4\times {10}^{10}$ FLOPs, more than the $6N=7.8\times {10}^9$ of train­ing a 1.3B stu­dent on it, at lower uti­liza­tion (com­puted, Chap­ter 13).

Model col­lapse

Train each model on its pre­de­ces­sor's output and "tails of the orig­i­nal con­tent dis­tri­bu­tion dis­ap­pear" [216]: a rare event of prob­a­bil­ity $p$ is absent from $M$ sam­ples with prob­a­bil­ity $(1-p{)}^M$, and once missed it cannot come back. But accu­mu­lat­ing syn­thetic data along­side the real data, instead of replac­ing it, avoids col­lapse [217]. In linear regres­sion with $n_{\text{feat}}$ fea­tures, $n_{\text{samp}}$ sam­ples per gen­er­a­tion and noise vari­ance ${\sigma }^2$, the test error after $g$ gen­er­a­tions, in units of one gen­er­a­tion's error ${\text{Err}}_1$, is

because under accu­mu­la­tion gen­er­a­tion $i$'s new noise is only $1/i$ of the data. With $n_{\text{feat}}=10$, $n_{\text{samp}}=60$ and $\sigma =1$, one gen­er­a­tion's error is $10/49=0.204$. By $g=10$, replac­ing reaches 2.04 and accu­mu­lat­ing 0.316, under the bound of 0.336 (com­puted). In another regres­sion set­ting, though, even 1% syn­thetic data can cause col­lapse [218].

Con­sent and licenses

The pool is also shrink­ing. From 2023 to 2024, new robots.txt restric­tions (a site's machine-read­able crawl per­mis­sions) fully restricted more than 5% of C4's tokens and more than 28% of its most crit­i­cal sources [219], and dataset hosts omit licenses more than 70% of the time [220]. Yet the 8 TB openly licensed Common Pile v0.1 trains 7B models com­pet­i­tive with Llama 1 and 2 7B [192]. These are public find­ings, not legal advice.

3.7 Tokenization: from text to integers

A trans­former reads inte­gers from a fixed set of size $V$. How does a string of Uni­code become that inte­ger sequence, and which choices in this step come back to bite you? Words leave an open vocab­u­lary and bytes make sequences four times longer, so sub­words are the middle ground [19]. A tok­enizer is a com­pres­sor with a fixed code­book, about 4 bytes per token in tik­to­ken [221]. That ratio sets FLOPs per unit of text (Chap­ter 1) and how much text a con­text holds, and it is frozen for the model's life.

Byte-pair encod­ing

Byte-pair encod­ing (BPE), a com­pres­sion method, was adapted by Sen­nrich et al. to text: write each word as char­ac­ters plus an end-of-word marker, then repeat­edly merge the most fre­quent adja­cent pair, never across words. The merge count is the only hyper­pa­ram­e­ter, and encod­ing replays the merges in order [222], so the ordered merge list is the tok­enizer.

vocab = all base symbols (bytes or characters)
words = {word: count} from the pre-tokenized corpus
repeat until len(vocab) == V:
    pairs = count adjacent symbol pairs, weighted by word count
    (a, b) = most frequent pair
    replace every "a b" with "ab" in all words
    vocab.add("ab"); merges.append((a, b))
encode(text): pre-tokenize, split chunks into base symbols,
              apply merges in the order learned

Sen­nrich et al.'s toy dic­tio­nary is low 5 times, lower 2, newest 6, widest 3: 95 sym­bols over ten let­ters and _. The first three merges (9 occur­rences each) build the suffix est_. After 12 merges the vocab­u­lary has 23 sym­bols, the corpus is 22 tokens (4.32 times shorter), and the unseen lowest encodes as low + est_ (com­puted). In Figure 17, watch each merge's count fall from 9 to 3: com­pres­sion flat­tens as $V$ grows.

Byte-level BPE and the pre-tok­enizer

Uni­code char­ac­ters would need over 130,000 base sym­bols. Bytes need 256, so noth­ing is ever unknown. GPT-2 also for­bids merges across char­ac­ter cat­e­gories except spaces, so words never fuse with punc­tu­a­tion [19]. A regex pre-tok­enizer cuts text into chunks (con­trac­tions, let­ters, digits, sym­bols, white­space, with an optional lead­ing space), and merges stay inside chunks [223]:

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

GPT-2's 50,257 tokens are 256 bytes, 50,000 merges and an end-of-text token [19, 224]. Merges can stop inside a multi-byte char­ac­ter, so byte vocab­u­lar­ies can emit invalid UTF-8 [225, 226]. Table 3.4 com­pares other designs. Uni­gram matches or beats BPE for masked models [227], yet decoder LLMs almost all use BPE, for rea­sons no public study set­tles.

Llama 3 adds 28K non-Eng­lish tokens to 100K from tik­to­ken, improv­ing com­pres­sion and down­stream results with no impact on Eng­lish [6]. Its config's 128,256 includes 256 reserved spe­cial tokens [24, 17].

Spe­cial tokens and bound­aries

Spe­cial tokens mark doc­u­ment ends and chat roles. Chat tem­plates and loss mask­ing are in Sec­tion 8.2. Never let user text become spe­cial tokens, which tik­to­ken refuses by default [238]. Serve the exact train­ing tem­plate, with­out dou­bled BOS tokens [239]. Reserve ids up front, as Llama 3 did [24]. Bound­aries bias models too: DeepSeek-V3 ran­domly splits merged punc­tu­a­tion-and-new­line tokens in train­ing [2], and exact byte-level scor­ing gains 18% on fill-in-the-middle coding [240]. Pre-tok­eniza­tion is a hyper­pa­ram­e­ter: SuperBPE's merges across spaces cut tokens by up to 33% [241], and long fine-tuning can jus­tify a tok­enizer swap [242].

Lan­guages and num­bers

A lan­guage's tok­eniza­tion pre­mium is its token count over Eng­lish for the same con­tent, on par­al­lel sen­tences [243]:

A window holds $1/\text{premium}$ as much con­tent, and atten­tion-score cost grows as ${\text{premium}}^2$ (Table 3.5). Their speak­ers pay more for worse results [244], Eng­lish-cen­tric tok­eniz­ers add up to 68% to mul­ti­lin­gual train­ing cost [245], and bytes are no cure: Burmese needs four times Chi­nese's bytes [243]. Train the tok­enizer on the pre­train­ing lan­guage mix.

Digits are the other trap. Left-to-right chunks split 1234567 as 123|456|7, but car­ries run from the right, where 1|234|567 aligns (com­puted). Right-to-left groups lifted GPT-3.5 and GPT-4 addi­tion from 75.6% to 97.8% with 8 shots, and left-to-right fell to 8.25% when a final carry length­ened the answer [246].

3.8 Vocabulary size, byte-level models and bits per byte

How big should the vocab­u­lary be, can we skip tok­eniza­tion alto­gether, and how do you com­pare two models whose tokens are dif­fer­ent? A bigger vocab­u­lary buys shorter sequences with dimin­ish­ing returns, and its costs matter most for small models.

Untied models have two such matri­ces. For Llama 3 8B that is $2\times \mathrm{128,256}\times \mathrm{4,096}=1.05$B param­e­ters (13.1%) and 6.5% of $2N$ [17], and Table 3.6 shows the share fall­ing roughly as $1/(n_{\text{layers}}d)$. So Kaplan et al. drop embed­dings from $N$, which "pro­duces sig­nif­i­cantly cleaner scal­ing laws" [25]. Logits take 3.91 GiB in FP32 per 8,192-token sequence (com­puted). Bigger vocab­u­lar­ies cut loss mostly on the 2,500 most fre­quent words while rare-tail loss rises [248].

Vocab­u­lar­ies are padded for fast mat­muls. OLMo 2's 100,352 and DeepSeek-V3's 129,280 are mul­ti­ples of 128 [21, 18], and nanoGPT says so out­right [37]:

vocab_size: int = 50304
# GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency

What $V$ buys is com­pres­sion, with gains that shrink expo­nen­tially [242]. Tao et al. fit tokens per char­ac­ter [249]:

It gives 3.95 char­ac­ters per token at 32K, 4.34 at 128K and 4.38 at 256K: 11% more for eight times the vocab­u­lary (com­puted). The fit turns near 220K to 230K, and it pre­dicts 9% fewer tokens from 32K to 128K against Llama 3's mea­sured 19.5% on other data. In Figure 18, slide $V$: the param­e­ter share col­lapses as models grow while com­pres­sion flat­tens.

At infer­ence size can pay. Llama 3 8B's 96,256 extra rows over 32K cost $2\times \mathrm{4,096}\times \mathrm{96,256}\times 2$ bytes = 1.47 GiB in BF16. At 128 KiB of KV cache per token ($2\times 32\times 8\times 128\times 2$ bytes), 19.5% fewer tokens repays them once about 61,600 old-tok­enizer tokens are cached (com­puted, after Dagan et al. [242]), rou­tine in long-con­text serv­ing (Chap­ter 13).

Scal­ing laws with vocab­u­lary

Tao et al. split param­e­ters into non-vocab­u­lary $N_{\text{nv}}$ and $N_{\text{v}}=Vd$, count data in char­ac­ters $H$, and use a uni­gram-nor­mal­ized loss, because a larger $V$ raises per-token loss by offer­ing more choices [249]:

A bigger $V$ shrinks $f(V)$ but grows $Vd$. Fit­ting ${ℒ}_u=-E+A_1/N_{\text{nv}}^{{\alpha }_1}+A_2/N_{\text{v}}^{{\alpha }_2}+B/D^{\beta }$, they find opti­mal vocab­u­lary param­e­ters grow as $N_{\text{nv}}^{0.83}$, and that Llama 2 70B "should have been at least 216K, 7 times larger than its vocab­u­lary of 32K". At 3B, 43K beat 32K on ARC-Chal­lenge (32.0 against 29.1) at equal FLOPs, one task at one scale [249]. The fit repro­duces their 35K, 60K and 218K optima (com­puted). See also Chap­ter 6.

Com­pres­sion alone is a weak pre­dic­tor. PathPiece, which seg­ments text into the fewest pos­si­ble tokens, did not give better down­stream models [250]. Yet an input vocab­u­lary 128 times larger, decou­pled from the output vocab­u­lary, let a 400M model match a 1B base­line's train­ing loss [251].

Byte-level models

Drop­ping the tok­enizer removes pre­mi­ums and glitch tokens but needs 3.4 to 4.4 times more posi­tions (com­puted). Patch models run a large global net­work once per patch of bytes (Table 3.7). Sub­words' edge may come from through­put and word bound­aries [252]. As of Novem­ber 2026 the lead­ing open models still ship byte-level BPE tok­eniz­ers, includ­ing DeepSeek-V3 [2] and Qwen3 [198].

Bits per byte

Per-token loss and per­plex­ity are not com­pa­ra­ble across tok­eniz­ers. Bits per byte (BPB) is: total code length $n_{\text{tok}}ℒ$ nats, divided by $\ln 2$ for bits and by the text's $n_{\text{byte}}$ UTF-8 bytes [189].

With GPT-2's tok­enizer the Pile has 0.29335 tokens per byte [189], so 2.5 nats per token is 1.058 BPB (com­puted). DeepSeek-V3 reports Pile BPB [2]. Count bytes on the unnor­mal­ized text, give spe­cial tokens zero bytes, and com­pare only on iden­ti­cal text.

3.9 Testing data decisions at small scale

A full run costs mil­lions of dol­lars. How do you decide with small runs that dataset A beats dataset B, and how sure can you be? Train small models that differ only in data, and treat their scores as noisy mea­sure­ments.

DCLM's rank­ings at 400M, 1B and 3B cor­re­late with 7B at $r=0.838$, 0.956 and 0.982, and a 1B run costs 240 H100-hours against 3,700 [167]. In DataDecide's 1,050 models (25 recipes, 14 sizes, 3 seeds), rank­ing at 150M picked the 1B winner in about 80% of pairs for $0.15\times 15/(1\times 100)=2.25\%$ of its com­pute (com­puted), no scal­ing-law method did better, and like­li­hood met­rics made MMLU, ARC, HellaSwag, MBPP and HumanEval over 80% pre­dictable [258]. High signal-to-noise bench­marks decide better [259].

Small from-scratch runs score at chance on MMLU [208], so value a small dataset by anneal­ing a partly trained model on a mix con­tain­ing it. Llama 3 anneals a 50%-trained 8B model on 40B tokens, 30% new data, for $40\text{B}/15\text{T}=0.27\%$ of the run (com­puted). In a sep­a­rate test, anneal­ing a pre-trained 8B model with the GSM8K and MATH train­ing sets in the mix raised its val­i­da­tion scores by 24.0% and 6.4%, but the gain at 405B was neg­li­gi­ble [6]: effects can shrink with scale (Chap­ter 7). Swarm regres­sions serve mix­tures (Sec­tion 3.5).

Seed-to-seed noise in bench­mark scores is mea­sured in Sec­tion 10.9. DataDecide finds up to 2 points at 1B [258]. With noise ${\sigma }_{\text{seed}}$ per run and $n_{\text{seed}}$ seeds per arm, a true gap $\Delta$ and recipes whose true scores spread by ${\sigma }_{\text{true}}$ give ($\Phi$ is the stan­dard normal cumu­la­tive dis­tri­bu­tion func­tion):

The last, the expected share of cor­rectly ordered pairs, is derived in this guide (sign agree­ment of cor­re­lated Gaus­sians): 0.852 when the spread is twice the noise, 0.75 when equal (com­puted, matched by sim­u­la­tion). DataDecide's metric is this share, near Kendall's $\tau$ rescaled to 0⁠–⁠1 [258].

Defaults: like­li­hood met­rics or bits per byte at small scale, aver­aged final check­points, two or three seeds when the gap is under 2 points, anneals for small sources, one con­fir­ma­tion at a larger scale, and data checks before launch. Con­tam­i­na­tion, includ­ing GPT-3's fil­ter­ing bug, is in Sec­tion 10.5.

To try it, use FineWeb-Edu's 10B-token sample [169] and build-nanogpt, about an hour and $10 for GPT-2 124M [260]. $6ND=6\times 1.244\times {10}^8\times {10}^{10}=7.47\times {10}^{18}$ FLOPs on 8 H100s at 989.5 TFLOP/s [4] and 40% MFU (illus­tra­tive) is 0.66 hours, so three seeds per arm for two arms take 3.9 hours (com­puted).

DCLM's AskLLM filter agreed with human anno­ta­tors far better than its fastText fil­ters (about 82% against 73% ROC-AUC). Which built the better train­ing set?

The fastText fil­ters: over 31 Core against about 28.5 at 1B-1x [167]. Human agree­ment does not pre­dict use­ful­ness: judge a filter by train­ing on what it keeps.

FineWeb's MinHash uses 14 bands of 8 hashes. How often are doc­u­ments at Jac­card sim­i­lar­ity 0.8 flagged, and what do 450 bands of 20 change?

$1-(1-{0.8}^8{)}^{14}=0.92$, against $1-(1-{0.8}^{20}{)}^{450}=0.995$, while at 0.7 the chance falls from 0.56 to 0.30 (com­puted). More hashes sharpen the thresh­old at about 80 times the cost [5, 168].

A team plans 15T tokens for a 70B model using only FineWeb-Edu (score at least 3). What goes wrong, and what are two fixes?

FineWeb-Edu has 1.3T tokens, so that is 11.5 epochs, worth about 8.6 times the unique data under Muen­nighoff's law (com­puted). Lower the thresh­old (5.4T tokens at score 2), ensem­ble clas­si­fiers as Nemotron-CC did, or add rephrased syn­thetic data [5, 209, 174].

Model A (32K vocab­u­lary) reaches 2.1 nats per token and model B (128K) 2.3 on the same text. Which is better?

Con­vert to bits per byte first. At 3.5 and 4.4 bytes per token (illus­tra­tive), A is at $2.1/(\ln 2\times 3.5)=0.866$ BPB and B at 0.754 (com­puted), so B is better despite its higher loss [249, 189].

A 150M abla­tion shows recipe A beat­ing B by 0.8 points on HellaSwag, one seed each, with 1 point of seed noise. Ship A?

Not yet. The gap's stan­dard devi­a­tion is $2^{1/2}=1.41$, so with no true dif­fer­ence a gap of 0.8 either way hap­pens $2(1-\Phi (0.57))=57\%$ of the time (com­puted). Use like­li­hood met­rics or more seeds, since HellaSwag also needs a com­pute thresh­old before it pre­dicts well [258].

4. Architecture and the pretraining objective

The objec­tive gives the model a prob­lem; the archi­tec­ture gives it a way to com­pute an answer. Work through a Llama 3 layer with its actual shapes, and keep track of which oper­a­tions con­sume param­e­ters, FLOPs and memory.

The objec­tive is one line: pre­dict the next token, scored by cross-entropy, at every posi­tion of every train­ing sequence (Sec­tion 4.1). The func­tion is one family: the decoder-only Trans­former, a stack of iden­ti­cal blocks that each read from and add to a run­ning vector per token. Llama 3 "uses a stan­dard, dense Trans­former archi­tec­ture", and its authors write that "our per­for­mance gains are pri­mar­ily driven by improve­ments in data qual­ity and diver­sity as well as by increased train­ing scale." [6]

So the diff from 2017 is small in code but not in con­se­quences. A few edits decide how fast the model trains, what each gen­er­ated token costs in memory and whether a long run sur­vives. We take one Llama 3 8B block apart (Sec­tion 4.2), then its sta­bil­ity guards (Sec­tion 4.3) and the evi­dence for each edit (Sec­tion 4.4). Sec­tions 4.5 to 4.8 cover the big depar­tures: atten­tion and the key-value (KV) cache, posi­tions, mix­ture of experts and recur­rent layers. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. Full FLOP and memory account­ing is in Chap­ter 1, tok­eniz­ers in Chap­ter 3, opti­miza­tion in Chap­ter 5.

4.1 The objective: predicting the next token

What does pre­train­ing min­i­mize, how should you read the number in the train­ing log, and why did the causal decoder-only model become the default? A lan­guage model assigns a prob­a­bil­ity to every pos­si­ble next token. Train­ing raises the prob­a­bil­ity of the token that actu­ally came next, at every posi­tion at once.

A doc­u­ment becomes token ids $x_1,\dots ,x_s$ from a vocab­u­lary of $V$ entries. A causal mask lets posi­tion $t$ see only $x_{\le t}$. The final hidden state $x_t^{(L)}$ times an output matrix $W_{\text{out}}\in {ℝ}^{V\times d}$ gives logits $y_t$, and a soft­max with nor­mal­izer $Z_t={\sum }_j{e}^{y_{t,j}}$ turns them into prob­a­bil­i­ties $p_t$. GPT-1 stated the objec­tive this way [261]. The loss is the mean neg­a­tive log prob­a­bil­ity of the true next token:

One for­ward pass yields $s-1$ train­ing sig­nals. The second form shows that adding one con­stant to every logit changes noth­ing, which mat­ters in Sec­tion 4.3.

Read­ing the number

Logs report $ℒ$ in nats per token. Divide by $\ln 2$ for bits. Per­plex­ity, $e^{ℒ}$, is the effec­tive number of equally likely next tokens, and it is expo­nen­tial: a 0.1-nat drop is a 9.5% drop in per­plex­ity, since $e^{-0.1}=0.905$ (com­puted).

Per-token losses depend on the tok­enizer. Bits per byte, which com­pares losses across tok­eniz­ers, is defined in Sec­tion 3.8. The Pile authors prefer it "due to its invari­ance to dif­fer­ent tok­eniza­tion schemes" [189]. Llama 3's tok­enizer improves Eng­lish com­pres­sion "from 3.17 to 3.94 char­ac­ters per token" over Llama 2's [6]. Treat­ing char­ac­ters as bytes, 1.80 nats per token is $1.80/(\ln 2\times 3.94)=0.659$ bits per byte with the new tok­enizer and 0.819 with the old, while per­plex­ity reads 6.05 in both (com­puted).

At step 0 the model pre­dicts roughly uni­formly, so the loss should start near $\ln V$: 11.76 nats for Llama 3's 128,256 rows [17] (com­puted). The ln V check at step 0 is in Sec­tion 12.5. A first loss far above it means large, con­fi­dently wrong logits. PaLM, whose input and output embed­dings are shared, scales its output logits by $1/\sqrt{d}$ to avoid this [34].

Why the causal decoder-only model won

It was not obvi­ous in 2019. T5 reported under fine-tuning that "For all tasks, the encoder-decoder archi­tec­ture with the denois­ing objec­tive per­formed best", with a GLUE aver­age (a suite of lan­guage-under­stand­ing tasks) of 83.28 against 73.78 for a decoder-only lan­guage model (reported) [177]. Zero-shot use flipped the answer. At 5B param­e­ters, Wang et al. found that "causal decoder-only models trained on an autore­gres­sive lan­guage mod­el­ing objec­tive exhibit the strong­est zero-shot gen­er­al­iza­tion after purely unsu­per­vised pre­train­ing" [262]. The eco­nom­ics agree. A full lan­guage-mod­el­ing loss uses every token, while their masked setup scored "only on the tar­gets (18% of tokens on aver­age)" [262]. GPT-3 kept "the same model and archi­tec­ture as GPT-2" [36].

Dis­sent remains. UL2 finds that "the choice of the denoiser has sig­nif­i­cantly more impact" than the back­bone [263], and T5Gemma turns pre­trained decoders into encoder-decoders [264]. As of Novem­ber 2026, pre­train­ing still starts from a causal decoder. Whether next-token pre­dic­tion alone is the right objec­tive is open, and multi-token pre­dic­tion is in Sec­tion 4.8.

4.2 Inside a modern block: Llama 3 8B, operation by operation

What exactly hap­pens to a token inside one layer of Llama 3 8B, and where do its 8 bil­lion param­e­ters and its FLOPs go? Pic­ture the resid­ual stream: one vector of width $d$ per token that every block reads and adds back to. Atten­tion mixes infor­ma­tion across posi­tions. The MLP trans­forms each posi­tion on its own.

The 2017 Trans­former nor­mal­ized after each resid­ual add (post-norm) and used a ReLU MLP of width $4d$ [266]. GPT-2 moved layer norm "to the input of each sub-block" [19], PaLM adopted SwiGLU and dropped biases [34], and LLaMA added RMSNorm and rotary posi­tions (RoPE) [46]. Llama 3 adds grouped-query atten­tion (GQA) with 8 KV heads and a RoPE base of 500,000, and oth­er­wise "does not devi­ate sig­nif­i­cantly from Llama and Llama 2" [6].

The block in order

Llama 3 8B has $d=\mathrm{4,096}$, $h=32$ query heads, $h_{kv}=8$ KV heads of dimen­sion $d_h=128$, $d_{\text{ff}}=\mathrm{14,336}$ and 32 layers [6, 17]. For $s$ tokens, one block does this:

1. Nor­mal­ize the stream $x\in {ℝ}^{s\times 4096}$: $u=\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(x)$, with $ϵ={10}^{-5}$ [17].
2. Project with­out biases: $q=u{W}_q$ to $32\times 128$, and $k=u{W}_k$, $v=u{W}_v$ to $8\times 128$ each [17].
3. Rotate $q$ and $k$, never $v$, with RoPE (Sec­tion 4.6).
4. Score $q{k}^{\top }/\sqrt{128}$ per head, each KV head serv­ing $32/8=4$ query heads, under a causal mask and a mask between packed doc­u­ments [6].
5. Soft­max, take the weighted sum of $v$ and con­cate­nate heads back to $s\times 4096$.
6. Project with $W_o$ and add: $x^{\prime }=x+\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}(\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(x))$.
7. Nor­mal­ize, $u^{\prime }=\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(x^{\prime })$, then gate $g=u^{\prime }{W}_{\text{gate}}$ and up $a=u^{\prime }{W}_{\text{up}}$, each to 14,336.
8. Form $\mathrm{S}\mathrm{i}\mathrm{L}\mathrm{U}(g)\odot a$, where $\mathrm{S}\mathrm{i}\mathrm{L}\mathrm{U}(z)=z\sigma (z)$ and $\odot$ is the ele­men­t­wise prod­uct, project down with $W_{\text{down}}$ and add: $\text{out}=x^{\prime }+\mathrm{M}\mathrm{L}\mathrm{P}(\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(x^{\prime }))$.

After 32 blocks come a final RMSNorm and an untied $\mathrm{128,256}\times \mathrm{4,096}$ output matrix [17]. The 128,256 rows are the paper's 128,000 tokens plus 256 reserved spe­cial tokens [24]. Figure 19 walks these oper­a­tions with their shapes, param­e­ters and FLOPs. Drag the con­text slider and watch the atten­tion-score seg­ment of the FLOPs bar grow while the param­e­ter bar stays still.

Where the param­e­ters go

The param­e­ter count is derived in Sec­tion 1.1 (Table 1.1). From the shapes in Table 3 of the Llama 3 paper and the released con­fig­u­ra­tion, Llama 3 8B has 218,112,000 param­e­ters per block and 8,030,261,248 in total, or 8,028,164,096 with 128,000 rows (com­puted) [6, 17]. The SwiGLU MLP is 80.8% of each block (com­puted). Atten­tion takes only 19.2% because its 8 KV heads make K and V a quar­ter the width of Q. With multi-head atten­tion (MHA, one KV head per query head) it would be 27.6% (com­puted). The same count gives 70,553,706,496 for 70B and 405,853,388,800 for 405B (com­puted).

Where the FLOPs go

A for­ward weight matmul costs 2 FLOPs per param­e­ter per token, and the embed­ding lookup costs noth­ing. Scores and the weighted sum of values cost $4h{d}_h$ FLOPs per layer for each key attended, and under a causal mask the aver­age token attends to about $s/2$ keys:

At Llama 3 8B's shapes, the atten­tion scores and the weighted sum of values take 3.5% of for­ward FLOPs per token at 2,048 tokens, 12.5% at 8,192, 36% at 32,768 and about 70% at 131,072 (com­puted from the Llama 3 paper's Table 3 [6], count­ing a causal mask as an aver­age of about $s/2$ keys per token and the weight mat­muls, includ­ing the output layer, at 2 FLOPs per param­e­ter). At 8,192 tokens that is $1.50\times {10}^{10}$ for weights plus $32\times 4\times \mathrm{4,096}\times \mathrm{4,096}=2.15\times {10}^9$ for atten­tion, and the two terms meet near 57K tokens (com­puted). Train­ing costs about three times the for­ward pass, $5.15\times {10}^{10}$ FLOPs per token at 8K (com­puted). Sec­tion 1.2 owns the budget. Here the point is that "6N per token" hides a con­text term that dom­i­nates at 128K.

SwiGLU and the 2/3 cor­rec­tion

SwiGLU replaces the MLP's first matrix and non­lin­ear­ity with a gated prod­uct [26, 267]:

A gated linear unit (GLU) layer mul­ti­plies one pro­jec­tion by a gated copy of another, so it has three weight matri­ces instead of two, and Shazeer cut its hidden width by a factor of 2/3 to keep param­e­ters and com­pu­ta­tion equal to the orig­i­nal [26]. Against a $4d$ MLP, that parity width is

OLMo 2 uses about $8d/3$ rounded up to the near­est mul­ti­ple of 128, which is 11,008 for its 7B model [7]. Llama 3's widths of 14,336, 28,672 and 53,248 are $3.5d$, $3.5d$ and $3.25d$, so they sit 31%, 31% and 22% above parity (com­puted from the Llama 3 paper's Table 3) [6]. Meta chose to spend more in the MLP than the 2/3 rule asks.

Tying the input embed­ding to the output layer saves one $V\times d$ matrix: 31% of GPT-2 small's 124,439,808 param­e­ters [20] but 0.5% of Llama 3 405B (com­puted). Press and Wolf rec­om­mend tying [269]. In prac­tice small models tie and large ones untie (Table 4.1 in Sec­tion 4.4).

4.3 Normalization and the stability guards

Where do the norms go, which parts of a trans­former blow up in long runs, and which archi­tec­tural guard addresses each fail­ure? Pic­ture the resid­ual stream with two soft­maxes in it: atten­tion inside every block and the output at the top. Each fails in its own way [270, 271], and norm place­ment sets how fast the stream grows. Why loss spikes happen is explained in Sec­tion 5.5.

Pre-norm, post-norm and RMSNorm

Post-norm nor­mal­izes the stream after each resid­ual add. Pre-norm nor­mal­izes only each sub­layer's input, which leaves an iden­tity path from output to input. At ini­tial­iza­tion, post-norm's gra­di­ents near the output are of order $𝒪(d\sqrt{\ln d})$ what­ever the depth, and pre-norm's are much smaller. So post-norm needs warmup: with­out it, Xiong et al.'s Trans­former reached a BLEU score (n-gram over­lap with ref­er­ence trans­la­tions) of only 8.45, while pre-norm trained well with none [272].

Pre-norm has a cost. If each block adds a unit-vari­ance update to a unit-vari­ance stream, the stream norm grows like $\sqrt{L+1}$: 3.0 at 8 layers and 11.4 at 128 (illus­tra­tive). Deep blocks then nudge a long vector, so they matter less. That fits the claim that "nearly half of the layers are less effec­tive than expected" [273], and that you "can elim­i­nate up to roughly half of the layers" of Llama 2 70B [274].

RMSNorm drops LayerNorm's mean sub­trac­tion [275] and keeps a learned gain $g$:

Its authors report "com­pa­ra­ble per­for­mance against LayerNorm" with 7% to 64% less run­ning time [276]. OLMo 2's abla­tions "show no dif­fer­ence between the two" [7]. It is a speed win, not a qual­ity win.

Two soft­maxes, three guards

Atten­tion logits $⟨q_i,k_j⟩/\sqrt{d_h}$ can grow until "the atten­tion weights col­lapse to one-hot vec­tors" [270], called atten­tion entropy col­lapse [277]. ViT-22B diverged this way [278]. Output logits fail dif­fer­ently. Cross-entropy ignores a shift shared by every logit, so noth­ing anchors $\log Z$ and it can drift, a fail­ure Worts­man et al. see "in models with no weight decay regard­less of scale" [270].

QK-norm guards the atten­tion soft­max by nor­mal­iz­ing $q$ and $k$ per head before the dot prod­uct [278, 279]. It "allows train­ing a model with 1.2B param­e­ters at learn­ing rate 0.3" [270], and Qwen3 and Gemma 3 adopted it [198, 265]. But OLMoE kept it "despite it reduc­ing through­put by almost 10%" [280], one study found it worst on long-con­text eval­u­a­tions [281], and it "is not appli­ca­ble to multi-head latent atten­tion (MLA), because its Key matri­ces are not fully mate­ri­al­ized during infer­ence" [72]. DeepSeek nor­mal­izes the com­pressed latents instead [282], and Kimi K2 rescales a head's query and key weights when its max­i­mum logit passes 100 [72] (QK-Clip, Chap­ter 5).

z-loss guards the output soft­max by penal­iz­ing $\log Z$ directly:

When $\log Z>0$ it lowers every logit in pro­por­tion to its prob­a­bil­ity, the one direc­tion cross-entropy cannot see. Pub­lished z-loss coef­fi­cients run from ${10}^{-5}$ to ${10}^{-4}$. PaLM adds ${10}^{-4}·{\log }^{2}Z$ to its loss [34], as do Worts­man et al. [270]. Chameleon [271] and Olmo 3 [12] use ${10}^{-5}$. The OLMo 2 paper gives both values: ${10}^{-5}$ in its Table 1 and ${10}^{-4}$ in Sec­tion 3.3.3 [7]. In two 1B abla­tions it made no mea­sur­able dif­fer­ence to qual­ity [283, 13].

Soft-cap­ping bounds logits smoothly. Gemma 2 uses $c=50$ in atten­tion and $c=30$ at the output [268]:

With QK-norm it let an 830M model take 1.5 times the learn­ing rate of QK-norm alone [284]. Gemma 3 still dropped it for QK-norm [265]. One likely reason, which is our infer­ence: a capped score is not the plain soft­max that fused atten­tion ker­nels assume.

Where the norms are moving

Llama 3 and Qwen3 use pre-norm, $x^{\prime }=x+f(N(x))$ [6, 198]. OLMo 2 nor­mal­izes out­puts, $x^{\prime }=x+N(f(x))$, and Gemma 2 and 3 nor­mal­ize both, a pat­tern Kim et al. call Peri-LN [7, 268, 285]. OLMo 2's output norm and QK-norm "reduce the spike score of the gra­di­ents from 0.108 to 0.069 when applied together", and nei­ther helped alone [7]. Peri-LN beat pre-norm on C4 loss, aver­aged over five seeds, at all three sizes: 3.11 against 3.20 at the largest (reported) [285]. That evi­dence stops at 3.2B param­e­ters and 30B tokens.

PaLM dropped every bias and "found this to result in increased train­ing sta­bil­ity for large models" [34]. Qwen2.5 kept a QKV bias [286] that Qwen3 removed [198]. Newer options such as gated atten­tion [287] rest on one lab's evi­dence so far.

4.4 What survives ablation: evidence, shape and the recipes labs ship

Which of these choices are backed by evi­dence that trans­fers, how should depth and width be split, and what do cur­rent recipes actu­ally use? Most single changes move their metric by a few per­cent at most, and many do not sur­vive the move to another code­base. Narang et al. re-imple­mented dozens of Trans­former mod­i­fi­ca­tions in one code­base: "Sur­pris­ingly, we find that most mod­i­fi­ca­tions do not mean­ing­fully improve per­for­mance," and the ones that helped "were either devel­oped in the same code­base that we used or are rel­a­tively minor changes" [288]. Across ten archi­tec­tures, Tay et al. found that "the vanilla Trans­former has the best scal­ing behav­iour" [289]. Treat a gain mea­sured once, at one scale, by its inven­tors, as a hypoth­e­sis.

SwiGLU is the best-sup­ported tweak, and its gain is small. In Shazeer's T5-base runs, matched in param­e­ters and FLOPs, held-out log-per­plex­ity fell from 1.677 (ReLU) to 1.636 (SwiGLU), about 0.04 nats or 2.4% (com­puted), and the paper cred­its the result "to divine benev­o­lence" [26]. Narang et al. repro­duced it (final loss 1.838 to 1.789) [288]. Falcon's 1B abla­tion dis­sents: zero-shot accu­racy stayed at 49.2, and its Pile score (labeled ppl-pile, but below 1, so a log-scale loss rather than a per­plex­ity) moved from 0.895 to 0.891, "just at the thresh­old of being unlikely to char­ac­ter­ize an actual improve­ment", and MLP acti­va­tion memory rose 33%, so Falcon did not adopt it. Its authors add that "SwiGLU may require ded­i­cated hyper­pa­ram­e­ters tuning" [283]. Nemotron-4 and Nemotron-H use squared ReLU instead [290, 291, 292].

Figure 20 puts 36 pub­lished com­par­isons on one scale, the rel­a­tive change in each study's own metric, signed so that right is better (com­puted from reported values):

Count the dots inside the ±1% band, then find the long sticks. They come from single-lab tests, such as DeepSeek-V2's from-scratch atten­tion abla­tion (MQA at -16.2% on MMLU, mul­ti­ple-choice knowl­edge ques­tions described in Sec­tion 10.1, com­puted) [282]. Changes in dif­fer­ent met­rics com­pare in mag­ni­tude only.

Shape barely mat­ters for upstream loss. Kaplan et al. found that "aspect ratio in par­tic­u­lar can vary by a factor of 40 while only slightly impact­ing per­for­mance" [25]. It mat­ters more down­stream and in small models: Tay et al. find shape "highly impor­tant for down­stream trans­fer" [293], MobileLLM's 30- and 42-layer models beat 12-layer ones at about 125M [294], and Gemma 2's deeper 9B beat a wider one 52.0 to 50.8 [268]. GLM-4.5 went deeper because "deeper models exhib­ited better rea­son­ing capac­ity" [295]. Depth costs speed [288] and "increases LR sen­si­tiv­ity at a faster rate than scal­ing width" [270]. In prac­tice the width-to-depth ratio $d/L$ is 64 for GPT-2 small [20], 128, 102 and 130 for Llama 3 8B, 70B and 405B [6], 118 for DeepSeek-V3 [2] and 19 for MobileLLM-125M [294] (com­puted). Above about 1B param­e­ters, open models sit at 80 to 130.

Heads cost infer­ence FLOPs. Kimi K2 found that dou­bling them low­ered val­i­da­tion loss by only 0.5% to 1.2% but, at 128K con­text, "leads to an 83% increase in infer­ence FLOPs", so K2 uses 64 heads where DeepSeek-V3 uses 128 [72].

Table 4.1 lines up six open fam­i­lies on the choices in Sec­tions 4.2 to 4.6.

The con­sen­sus is plain: RMSNorm, a gated MLP and RoPE every­where, shared or com­pressed keys and values (OLMo 2 7B's MHA is the excep­tion), and, out­side Gemma, untied embed­dings from about 8B up. Labs differ on the sta­bil­ity guards.

4.5 Attention variants and the KV cache

Why did atten­tion change at all, how many key and value heads does a model need, and what do MQA, GQA, MLA and slid­ing win­dows each buy and cost? The pres­sure came from gen­er­a­tion. Each new token's query reads the stored keys and values of every ear­lier token, the KV cache, and each byte feeds only a few oper­a­tions. Shazeer showed that incre­men­tal multi-head atten­tion has a ratio of memory access to arith­metic of $\Theta (s/d+1/b)$ for con­text $s$, width $d$ and batch $b$, "caus­ing memory band­width to be a major per­for­mance bot­tle­neck" [74]. Batch­ing shrinks $1/b$, but $s/d$ is the cache itself. Each design below trades bytes against FLOPs and qual­ity.

In multi-head atten­tion (MHA) each of the $h$ query heads has its own keys and values. Multi-query atten­tion (MQA) shares "a single set of keys and values" across heads, which cut an incre­men­tal decoder step for a batch of 1,024 sequences on TPUv2 from 47 ms to 3.9 ms (reported) [74]. Grouped-query atten­tion (GQA) sits between, with $h_{kv}$ KV heads that each serve $h/h_{kv}$ query heads [75]. Each layer caches one key and one value vector per KV head [282]:

Figure 21 com­putes these caches for any shape. Pick fewer KV heads $h_{kv}$ and the cache falls in pro­por­tion. On the right, GQA and MLA caches grow with con­text while win­dowed and hybrid designs go nearly flat. Sec­tion 13.4 turns bytes into serv­ing capac­ity.

GQA cuts bytes, not FLOPs. Each query head still scores $s$ keys and sums $s$ values, so decode atten­tion's arith­metic inten­sity in BF16 rises from about 1 FLOP per KV byte (MHA) to $h/h_{kv}$: 8 for GQA-8, 64 for MQA at $h=64$ (com­puted). Decode atten­tion FLOPs equal the $2N$ weight FLOPs at

about 53,000 tokens for Llama 3 70B ($2\times 69.50\times {10}^9/(4\times 80\times 64\times 128)$, count­ing matmul param­e­ters only, com­puted). Only fewer query heads, win­dows or sparse atten­tion move that cross­over.

What shar­ing costs in qual­ity is con­tested, because the tests differ. DeepSeek trained three 7B dense models on 1.33T tokens, matched in param­e­ters by depth, and mea­sured 5-shot MMLU of 45.2 (MHA), 41.2 (GQA-8) and 37.9 (MQA) [282]. Ainslie et al. uptrained T5-XXL encoder-decoders for 5% of their pre­train­ing com­pute and report seven-task aver­ages of 47.2, 47.1 and 46.6, at 1.51 s, 0.28 s and 0.24 s per sample [75]. Gemma 2's 9B abla­tion gives a four-bench­mark aver­age of 50.3 with MHA and 50.8 with GQA [268]. The penalty looks small for uptrain­ing or mild group­ing and large from scratch on hard bench­marks. Shipped models accept GQA [6, 296, 198, 265, 28]. With fewer KV heads than tensor-par­al­lel ranks, heads must be repli­cated (Sec­tion 11.3).

MLA: cache a latent, not the keys

Multi-head latent atten­tion (MLA) com­presses instead of shar­ing [282]:

1. Down-project each hidden state to a latent $c_t^{KV}$ of width $d_c=512$ [2].
2. Up-project it to per-head keys $W^{UK}{c}_t^{KV}$ and values $W^{UV}{c}_t^{KV}$.
3. Cache only the latent. Since $q^{\top }k=((W^{UK}{)}^{\top }{W}^Q{x}_t{)}^{\top }{c}_s$ for hidden state $x_t$, $W^{UK}$ folds into $W^Q$ and $W^{UV}$ into $W^O$, so "we even do not need to com­pute keys and values out for atten­tion" [282].
4. RoPE would put a posi­tion-depen­dent rota­tion between $W^Q$ and $W^{UK}$, so "RoPE is incom­pat­i­ble with low-rank KV com­pres­sion" [282].
5. Decou­pled RoPE adds one rotary key of width $d_h^R=64$, shared by all heads and cached beside the latent [282].

That equals GQA "with only 2.25 groups" [282], or $61\times 576\times 2=\mathrm{70,272}$ bytes per token for DeepSeek-V3 (com­puted). In absorbed form all 128 heads do $2(576)+2(512)=\mathrm{2,176}$ FLOPs each per cached token on the same 1,152 bytes, about 242 FLOPs per byte (com­puted), against 8 for GQA-8 and close to an H100's dense BF16 ridge of 295 (Sec­tion 1.7). MLA decode atten­tion sits near the com­pute-bound regime, not deep in the memory-bound one. The qual­ity evi­dence is the authors' own: in DeepSeek-V2's MoE abla­tions MLA beat MHA on MMLU (50.0 against 48.7, 59.0 against 57.5) with 14% and 4% of its cache and slightly fewer acti­vated param­e­ters [282]. As of Novem­ber 2026 we found no inde­pen­dent dense test.

Win­dows and sinks

Slid­ing-window atten­tion lets each token see only the pre­vi­ous $w$ tokens, yet infor­ma­tion still trav­els $w$ tokens per layer. Mis­tral 7B's 32 layers with $w=\mathrm{4,096}$ give "a the­o­ret­i­cal atten­tion span of approx­i­mately 131K tokens", and a rolling buffer "reduces the cache memory usage by 8x" at 32K [296]. Gemma and gpt-oss inter­leave local and global layers (Table 4.1). With a frac­tion $f_g$ of global layers, the cache rel­a­tive to all-global is

Gemma 3's pat­tern ($f_g=1/6$, $w=\mathrm{1,024}$) keeps 0.193 of the cache at 32K and 0.173 at 128K (com­puted), with "min­i­mal" per­plex­ity impact [265]. Retrieval tests may dis­agree (Chap­ter 10).

Win­dows hide a trap: "a sur­pris­ingly large amount of atten­tion score is allo­cated to the ini­tial tokens, irre­spec­tive of their rel­e­vance". Keep­ing four of these sink tokens beside the window lets a model stream stably [297], and gpt-oss builds a sink into each head as "a learned bias in the denom­i­na­tor of the soft­max" [28]. If you serve with a window or evict cache entries, keep the first few tokens.

4.6 Position: RoPE and its alternatives

Atten­tion is blind to order. How does posi­tion get in, why did RoPE win, and what does its base fre­quency con­trol? With­out a mask or posi­tions, atten­tion is per­mu­ta­tion-equi­vari­ant: shuf­fle the tokens and each token's output is unchanged, only reordered. So posi­tion must be added. The 2017 Trans­former added sinu­soids, hoping they "may allow the model to extrap­o­late", and found learned embed­dings "nearly iden­ti­cal" [266]. ALiBi sub­tracts a per-head linear dis­tance penalty from the scores (slopes 1/2 to 1/256 for 8 heads), but its authors credit most of its edge to "improved avoid­ance of the early token curse" [298].

Rotary posi­tion embed­ding (RoPE) wants a score that depends only on con­tent and rel­a­tive offset [299]. Write one 2-D pair of a query as a com­plex number $z_q$ and rotate it by its posi­tion $m$, mul­ti­ply­ing by $e^{im\omega }$. For a key at $n$,

which depends on $m-n$ alone. A head of width $d_h$ has $d_h/2$ pairs, each at its own fre­quency ${\omega }_j$ with wave­length ${\lambda }_j$ in tokens:

Rota­tions pre­serve norms and $R_m^{\top }{R}_n=R_{n-m}$, so RoPE "encodes the abso­lute posi­tion with a rota­tion matrix and mean­while incor­po­rates the explicit rel­a­tive posi­tion depen­dency" [299]. It touches queries and keys, never values. Code builds no matrix: $q^{\prime }=q\odot \cos (m\omega )+\mathrm{r}\mathrm{o}\mathrm{t}(q)\odot \sin (m\omega )$, where $\mathrm{r}\mathrm{o}\mathrm{t}$ swaps and negates each pair.

The RoPE figure in Sec­tion 7.4 starts from this step and shows the whole spec­trum. Notice how little the slow­est pairs turn within a train­ing con­text.

The base sets those slow wave­lengths. With $d_h=128$, the slow­est pair has a wave­length of 54,410 tokens at base 10,000 and about 2.56M tokens at Llama 3's base of 500,000, and 15 of the 64 pairs have wave­lengths longer than 131,072 tokens (com­puted) [6]. These counts use the base alone. The released Llama 3.1 con­fig­u­ra­tion also rescales RoPE by a factor of 8 for wave­lengths beyond its orig­i­nal 8,192-token con­text [17], which stretches the slow­est wave­length to about 20.5M tokens and puts 25 pairs above 131,072 (com­puted). Pairs that never com­plete a turn within the con­text can carry con­tent nearly free of posi­tion, so long-con­text models raise the base: OLMo 2 to 500,000 [21], Qwen3 to 1,000,000 [198], Gemma 3 to 1M on global layers only [265]. Extend­ing RoPE beyond the trained length is Sec­tion 7.4.

What the fre­quen­cies do is con­tested. Against RoFormer's story of scores decay­ing with dis­tance [299], Bar­bero et al. found that trained Gemma 7B "greatly prefers to use the lowest fre­quen­cies of RoPE, which we sus­pect are used to carry seman­tic infor­ma­tion", and drop­ping the lowest 25% slightly improved 2B-model per­plex­ity (4.4414 against 4.4627) [300]. MiniMax-Text-01 rotates only half of each head [301], and MLA is par­tial RoPE by con­struc­tion. Models with no posi­tions (NoPE) still work, since the causal mask leaks order: they "acquire an implicit notion of abso­lute posi­tions" [302]. Cohere uses NoPE on its full-atten­tion layers [281], Llama 4 inter­leaves layers "with­out posi­tional embed­dings" [303], and recur­rent hybrids drop posi­tions entirely [304, 292] (Sec­tion 4.8). This guide's defaults as of late 2026: RoPE with a base of 500,000 to 1,000,000 for dense models, par­tial RoPE with MLA, and NoPE only in some global layers or in hybrids.

4.7 Mixture of experts: parameters without FLOPs

How can a model hold 671B param­e­ters but spend the FLOPs of about 37B per token, what keeps its experts busy, and what does spar­sity cost in memory and com­mu­ni­ca­tion? A mix­ture of experts (MoE) keeps many copies of the MLP, called experts, and a small learned router that sends each token to a few of them. Param­e­ters grow with the number of experts. FLOPs per token grow only with the experts each token visits. So DeepSeek-V3 stores 671B param­e­ters and acti­vates 37B per token (reported) [2].

Early mix­tures ran every expert and weighted them softly [305, 306]. Shazeer et al. made the gate sparse and named the two prob­lems every later paper fights: rout­ing col­lapse, where favored experts train faster and get chosen more, and the shrink­ing batch each expert sees [307]. Table 4.3 fol­lows the line to today.

Anatomy and param­e­ter count

The MoE replaces the MLP sub­layer, and atten­tion stays dense. A router $W_r\in {ℝ}^{d\times E}$ scores each token against $E$ routed experts (0.016% of a DeepSeek-V3 MoE layer, com­puted). Scores come from a soft­max (Mix­tral's over its top-$k$ logits [310]) or, in DeepSeek-V3, from sig­moids nor­mal­ized over the chosen experts [2]. Each token goes to its top $k$ experts, SwiGLU MLPs of width $d_e$ with $3d{d}_e$ param­e­ters, whose out­puts are gate-weighted and summed with any $E_s$ shared experts and the resid­ual. With $A$ atten­tion param­e­ters per layer, $L_{\text{moe}}$ sparse layers out of $L$, untied embed­dings and a final norm, the param­e­ters every token uses regard­less of rout­ing are

and the experts add the rest:

DeepSeek-V3 ($d=\mathrm{7,168}$, 61 layers, 3 dense, 58 MoE layers of 1 shared plus 256 routed experts of width 2,048, top-8) gives 671.03B and 37.55B: 37B (reported), about 37.6B by our count [2, 18]. Its routed experts are 97.4% of stored param­e­ters but 54% of active ones (com­puted). Con­ven­tions vary: gpt-oss counts the unem­bed­ding as active but not the embed­ding [28]. Expert-choice rout­ing, where experts pick tokens, bal­ances by con­struc­tion but peeks at future tokens [311].

Keep­ing experts busy

With­out bal­ance, rout­ing col­lapses and the busiest GPU sets the step time [2]. OLMoE with­out a bal­anc­ing loss left most first-layer experts as "dead weights" [280]. Switch's aux­il­iary loss is [30]

with $f_i$ the frac­tion of tokens dis­patched to expert $i$ (no gra­di­ent) and $P_i$ its mean router prob­a­bil­ity (dif­fer­en­tiable). At uni­form rout­ing $f_i=P_i=1/E$, so the loss is exactly $\alpha$ for any $E$ (derived). Too large an $\alpha$ adds inter­fer­ence gra­di­ents [313]. The hard limit is a capac­ity per expert of $\lfloor kT/E\times \text{CF}\rfloor$ slots for $T$ tokens and capac­ity factor CF, with over­flow tokens dropped to the resid­ual [30].

Drop­less ker­nels move imbal­ance from qual­ity to time and memory, because the busiest expert sets the step [315]. DeepSeek-V3 bal­ances with a bias ${\delta }_i$ that changes which experts are selected but not the gate values, so it adds no gra­di­ent [313, 2]:

where $c_i$ is expert $i$'s load and $\overline{c}$ the mean (DeepSeek writes $b_i$ and $\gamma$). V3 used $u=0.001$ for 14.3T tokens and 0 for the last 500B, and dropped no tokens [2]. Yet its own 1B abla­tion gives val­i­da­tion losses of 2.258 with a sequence-wise aux­il­iary loss, 2.253 with the bias method and 2.253 with a batch-wise aux­il­iary loss, and the authors con­clude that the key dif­fer­ence "lies in their bal­anc­ing scope" [2]. Bal­ance scope, more than drop­ping the loss, explains most of the gain.

Figure 22 walks one layer through rout­ing, capac­ity, drop­less exe­cu­tion, bias updates, scope and expert place­ment. Watch over­flow appear at CF 1.0 under equal expected loads.

The router z-loss [314],

is added with coef­fi­cient $c_z=0.001$. For 32 experts $c_z{L}_z$ is about 0.016 at unit-scale logits and about 1.8 at scale 20 (com­puted), so it bites only when logits grow. Insta­bil­ity tracked com­pute per token: Switch-XXL, with about 7x the FLOPs per sequence of Switch-C (6.3T against 890B in its Table 9, "nearly 10x" in the text), was the unsta­ble one [30].

The modern recipe and its costs

Fine-grained experts split each expert into smaller ones at equal param­e­ters and FLOPs [317], so top-8 of 256 gives $\left(\genfrac{}{}{0ex}{}{256}{8}\right)\approx 4.1\times {10}^{14}$ com­bi­na­tions (com­puted). V3's gran­u­lar­ity $g_{\text{moe}}=d_{\text{ff}}/d_{\text{expert}}$ [318] is $\mathrm{18,432}/\mathrm{2,048}=9$ (com­puted). Shared experts are con­tested. Dis­abling DeepSeekMoE's shared expert at eval­u­a­tion, in a model trained with it, raised Pile loss from 1.808 to 2.414 [317], a knock­out rather than a train­ing abla­tion. OLMoE found a shared expert slightly worse [280] and Qwen3 dropped them [198]. Expert par­al­lel­ism and DeepSeek-V3's node-lim­ited rout­ing are in Sec­tion 11.6.

Spar­sity saves FLOPs, not bytes. If $T$ decode tokens each pick $k$ of $E$ experts inde­pen­dently, a step reads this share of the routed experts:

For V3 that is 3.1% at $T=1$ and 87% at 64, about 568 GB of FP8 expert weights per step (com­puted, an upper bound). So V3 decodes on 320-GPU units, still memory-bound [2], and gpt-oss quan­tizes its experts to MXFP4 to fit one 80 GB GPU [28]. Serv­ing an MoE, includ­ing the 14.76 ms all-to-all floor for DeepSeek-V3, is in Sec­tion 13.6. Figure 23 com­putes them for any con­fig­u­ra­tion: slide the decode tokens and watch the expert share climb while train­ing FLOPs stay put.

Upcy­cling a dense check­point into an MoE pays off only when the addi­tional train­ing budget is small [319, 320]. MoE scal­ing laws are in Sec­tion 6.6.

4.8 Beyond softmax attention: recurrent layers, hybrids and new objectives

Can the KV cache be replaced by a fixed-size state, what does that cost, and are objec­tives beyond next-token pre­dic­tion worth using? Yes, at a price in recall, which a few atten­tion layers buy back. Multi-token pre­dic­tion is the one new objec­tive in pro­duc­tion.

Replace soft­max atten­tion's weight $\exp (q_t^{\top }{k}_s)$ by $\varphi (q_t{)}^{\top }\varphi (k_s)$, and asso­cia­tiv­ity turns atten­tion into a recur­rence [321]:

A 128 x 128 head state is 32 KiB in BF16 at any con­text, the KV of 64 tokens of one GQA head (com­puted). The price is capac­ity. Store $n_{\text{pairs}}$ random unit key-value pairs addi­tively and read one back: every other pair adds noise of vari­ance about $1/d_k$, so

0.90 at 16 pairs and 0.24 at 1,024 for $d_k=64$, matched by Monte Carlo (com­puted). Soft­max atten­tion keeps every key instead [322]. The fixes are decay and the delta rule, which over­writes the value stored for $k_t$ (${\beta }_t$ is a write strength, not an Adam moment) [322, 323]:

The losses are about recall. Recall explained 82% of the gap between atten­tion and gated con­vo­lu­tions [329], and trans­form­ers learn copy­ing with 100x fewer sam­ples than SSMs [330]. In a matched 8B study Mamba-2 trailed by 17 five-shot MMLU points at 1.1T tokens and 1.37 at 3.5T, and Phone­book stayed hard at any token count [331].

So hybrids won (Table 4.6). About 8% atten­tion layers min­i­mized loss in one study [331], and another rec­om­mends 3:1 to 6:1 linear to full layers [332].

Multi-token pre­dic­tion (MTP) trains each posi­tion to pre­dict sev­eral future tokens. With par­al­lel heads, 13B models solved 12% more HumanEval prob­lems, but small models got worse [335]. DeepSeek-V3 adds one sequen­tial module (depth $D_{\text{MTP}}=1$, since $D$ means tokens) that shares the embed­ding and output head [2]:

with weight $\lambda =0.3$ for 10T tokens, then 0.1 (not the RoPE wave­length ${\lambda }_j$). It lifted HumanEval from 44.5 to 53.7 but cut MMLU from 67.5 to 66.6 [2], for about 4% more train­ing FLOPs (com­puted, approx­i­mate: the module adds about 1.6B active param­e­ters per token, one MoE block, a pro­jec­tion and the shared 0.93B output head, against 37.6B). Using MTP heads as a spec­u­la­tive draft is in Sec­tion 13.7. Dif­fu­sion models such as LLaDA drop the causal mask and the KV cache [336], a research direc­tion, not a default, as of Novem­ber 2026.

A model with a 262,144-token vocab­u­lary prints a loss of 14.9 nats at step 0. Is that fine?

No. Uni­form pre­dic­tion gives $\ln \mathrm{262,144}=12.48$ nats (com­puted), so the logits start con­fi­dently wrong. A usual cause is a tied output layer ini­tial­ized at $𝒩(0,1)$ with­out the $1/\sqrt{d}$ logit scale PaLM applies [34].

Why is Llama 3 8B's MLP width 14,336 rather than $4d=\mathrm{16,384}$?

SwiGLU has three matri­ces, so param­e­ter parity with a $4d$ MLP is $8d/3=\mathrm{10,923}$ [26]. Llama 3's 14,336 is $3.5d$, 31% above parity and a mul­ti­ple of 1,024 (com­puted from [6]): a choice to put more param­e­ters in the MLP.

Llama 3 70B has 64 query heads and 8 KV heads. By what factor does GQA cut decode atten­tion FLOPs rel­a­tive to MHA?

None. Every query head still scores $s$ keys. GQA cuts KV bytes 8x (2.5 MiB to 320 KiB per token) and so lifts decode arith­metic inten­sity from about 1 to 8 FLOPs per byte (com­puted from [6, 75]).

A Switch-style layer has 64 experts, top-1, 4,096 tokens per group and capac­ity factor 1.0. The router is per­fectly bal­anced in expec­ta­tion. Do you drop tokens?

Yes, about 4.9% (com­puted from the bino­mial with the Switch for­mula [30]), because loads fluc­tu­ate around 64 and every expert above it over­flows. CF 1.25 drops 0.13% at the price of 20% pad­ding. Bal­ance in expec­ta­tion is not bal­ance in every step, one reason drop­less ker­nels won [315].

An 8B pure Mamba-2 model matches a Trans­former on aver­age zero-shot accu­racy. Which two eval­u­a­tions do you run before switch­ing, and what does the evi­dence pre­dict?

Five-shot MMLU and a retrieval test such as Phone­book. The matched 8B study above pre­dicts a 17-point MMLU gap at 1.1T tokens and Phone­book fail­ures at any scale [331]. Prefer a hybrid with a few atten­tion layers.

5. Optimization and training dynamics

An opti­mizer turns pre­dic­tion errors into param­e­ter updates. The learn­ing rate, batch size, ini­tial­iza­tion and number format deter­mine whether those updates improve the model, waste the budget or end the run. A useful loss curve has to be read along­side those choices.

You have an archi­tec­ture (Chap­ter 4) and a budget of about $6ND$ FLOPs (Sec­tion 1.2). Which opti­mizer set­tings spend it well, and how do they inter­act? Look at two recipe cards. Llama 3 405B used AdamW at a peak learn­ing rate of $8\times {10}^{-5}$, an 8,000-step linear warmup and a cosine decay to $8\times {10}^{-7}$ over 1.2M steps, with a batch that grew from 4M to 16M tokens. The team "observed few loss spikes and did not require inter­ven­tions to cor­rect for model train­ing diver­gence" [6].

DeepSeek-V3 keeps the core, AdamW with ${\beta }_1=0.9$, ${\beta }_2=0.95$, weight decay 0.1 and clip­ping at 1.0. But it holds the rate at $2.2\times {10}^{-4}$ for 10T tokens before decay­ing, grows the batch to 15,360 sequences of 4,096 tokens (62.9M, com­puted), and runs matrix mul­ti­plies in FP8 within 0.25% of a BF16 base­line's loss (reported), with no irrecov­er­able loss spikes [2].

Every number on these cards has a reason, and the num­bers are cou­pled. Chap­ter 11 covers how the batch is assem­bled across accel­er­a­tors and Chap­ter 12 keep­ing the job alive. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

5.1 AdamW: the update and what each knob does

What does one AdamW step com­pute, and why does each hyper­pa­ram­e­ter take the value it does in public recipes? Gra­di­ent descent moves a weight with 100 times larger gra­di­ents 100 times fur­ther. Adam keeps run­ning means of the gra­di­ent and of its square for every param­e­ter and divides the first by the root of the second [337]. Each coor­di­nate then moves by about $\eta$ what­ever its gra­di­ent scale, so one learn­ing rate serves embed­dings, atten­tion and MLP weights.

With $g_t$ the gra­di­ent at step $t$, $\theta$ the weights and every oper­a­tion ele­men­t­wise, one step in PyTorch's form is

The moments start at zero, so they are "biased towards zero, espe­cially during the ini­tial timesteps" [337]. Bias cor­rec­tion undoes this: after 10 steps $1-{0.95}^{10}=0.401$ but $1-{0.999}^{10}=0.010$ (com­puted). On step one the ratio is $g_1/\left|g_1\right|=\pm 1$, so every weight moves by about $\eta$ in a sign pat­tern set by one batch (com­puted), one reason to warm up. Each moment aver­ages about $1/(1-\beta )$ steps: 10 at 0.9, 20 at 0.95, 1,000 at 0.999. Table 5.1 com­pares recipes.

Weight decay and its two mean­ings

Under SGD an L2 penalty equals weight decay. Under Adam it does not, because the L2 term enters $g_t$ and is divided by ${\widehat{v}}_t^{1/2}$, so AdamW applies the decay out­side that divi­sion [338]. The paper's inde­pen­dent form leaves $\lambda$ unscaled. PyTorch and Optax mul­ti­ply it by $\eta$, though the inde­pen­dent form "reduces LR sen­si­tiv­ity" [270]. LLM reports mean the cou­pled form: in OLMo 2 "every param­e­ter is mul­ti­plied by $1-(0.1·lr)$ at every step" [7].

The cou­pled step is an expo­nen­tial moving aver­age (EMA):

so the weights aver­age the updates of roughly the last

steps, with $T$ the total steps and a con­stant batch $B$ in the second form. It is "a rough timescale over which updates are aver­aged" [339], and its best value in epochs stays roughly con­stant across model and data sizes [340]. Train $K$ times longer at the same $\eta$ and divide $\lambda$ by $K$. Power Lines fit $(\tau /T{)}_{\text{opt}}=1.084·{\mathrm{T}\mathrm{P}\mathrm{P}}^{-0.527}$, with TPP the tokens per param­e­ter, on μP models (Sec­tion 5.4) at 20 to 1,280 TPP [341].

Why ${\beta }_2=0.95$

No public con­trolled abla­tion at LLM scale set­tles this. A 20-step second moment tracks shifts in gra­di­ent scale. A stale 1,000-step one meet­ing a large gra­di­ent gives an over­sized update, a fail­ure "more likely to be observed in the train­ing of a deep model with a large batch size" [342]. Large batches also make $g_t^2$ less noisy, and 0.95 "is sub­op­ti­mal at smaller batch sizes (128 and below)" [27]. Marek et al. trained stably down to batch size one by hold­ing the half-life fixed in tokens: 0.95 at 512 sequences becomes ${0.95}^{64/512}=0.9936$ at 64 (com­puted) [343]. Later recipes copied GPT-3 and Llama 2 [36, 229]. PaLM, wor­ried about rare tokens, used ${\beta }_2=1-k^{-0.8}$ at step $k$ [34].

Epsilon and clip­ping

For a steady gra­di­ent RMS $g$, Adam's step shrinks by about $g/(g+ϵ)$ (Table 5.3). OLMo 2 cut $ϵ$ from ${10}^{-5}$ to ${10}^{-8}$, and "the gra­di­ent norm set­tles much more quickly and remains per­ma­nently lower" [7]. At Worts­man et al.'s largest scale the gra­di­ent RMS reached the default $ϵ$, and ${10}^{-15}$ improved loss [270] (Sec­tion 5.5).

Clip­ping takes one norm over all param­e­ters, after the data-par­al­lel all-reduce, and rescales when it exceeds $c$:

With $c=1$, a norm of 0.8 passes, 2.5 is scaled by 0.4 and 40 by 0.025 (com­puted), direc­tion kept. First a fix for explod­ing gra­di­ents in recur­rent net­works [344], clip­ping is set to 1.0 in Table 5.1, and OPT dropped to 0.3 early for sta­bil­ity [345]. A late rise of the norm in long decay­ing runs comes from "an unin­tended inter­ac­tion between weight decay, nor­mal­iza­tion layers, and the learn­ing rate sched­ule" [346].

Rivals rarely beat AdamW: apart from SGD, the opti­miz­ers Zhao et al. com­pared "all per­form com­pa­ra­bly" [347] (Sec­tion 5.7). Defaults: ${\beta }_1=0.9$, ${\beta }_2=0.95$ at mil­lion-token batches (higher under about 128 sequences), $ϵ\le {10}^{-8}$, cou­pled decay 0.1 but none on embed­dings, where it raises the gra­di­ent norm [7], and a clip of 1.0. Log the pre-clip norm and the share of steps clipped. Typ­i­cal mis­takes: $ϵ$ above the gra­di­ent RMS, and mixed decay con­ven­tions.

5.2 Warmup and learning-rate schedules

Why does nearly every run ramp the learn­ing rate up from zero and then bring it down, and does the shape of the descent matter at equal com­pute? Early, a large rate trav­els far from the ini­tial­iza­tion. Late, gra­di­ent noise makes the weights bounce across a valley by an amount that grows with $\eta$, and low­er­ing $\eta$ shrinks the bounce. The sched­ule sets when to travel and when to settle. Warmup is a sep­a­rate job: sur­viv­ing the first steps.

Warmup

Llama 3 405B warms up over 8,000 steps [6], Llama 2, OLMo 2, DeepSeek-V3 and SmolLM2 over 2,000 [229, 7, 2, 197]. Four expla­na­tions com­pete (Table 5.4). RAdam counts the effec­tive sam­ples in Adam's second moment and damps the step until ${\rho }_t$ exceeds 4:

The num­bers pick a winner. LLM warmups of 2,000 to 8,000 steps are 50 to 200 times the 40-step rule (com­puted), too long for the vari­ance story. Cur­va­ture fits: the pre­con­di­tioned sharp­ness "typ­i­cally starts off at high values (even in the large batch case)" [350]. OLMo 2 repro­duced its early spikes "by mainly reduc­ing the warmup period" [7]. For small scal­ing runs, Porian et al. warm up over $N$ tokens [27].

Decay shapes

With $x$ the post-warmup frac­tion of train­ing, cosine decay, from SGDR [352], is

usu­ally with ${\eta }_{min}=0.1{\eta }_{max}$. Linear decay to zero (D2Z) is ${\eta }_{max}(1-x)$ [339]. Warmup-stable-decay (WSD) holds the peak, then cools down over a short final stretch [353, 354], lin­early or with a 1-sqrt shape over the last $T_{\text{decay}}$ of $T$ steps (the paper writes $N$):

Sched­ule-free AdamW skips decay and eval­u­ates an aver­age of the iter­ates [355]. It won the self-tuning track of AlgoPerf 2024, a com­pe­ti­tion that times train­ing algo­rithms to a target on fixed work­loads, [356] but is sen­si­tive to the betas in LLM pre­train­ing [353].

What the evi­dence says

Area does not decide the final loss. WSD with a 20% cooldown aver­ages 0.90 of peak and cosine to 10% only 0.55 (Table 5.5), yet they end in the same place (Table 5.6). OLMo 2's authors say their result con­tra­dicts the folk wisdom that "area under the learn­ing curve mat­ters" [7]. Tissue et al.'s anneal­ing law is derived in Sec­tion 7.1.

A cosine fits one length: Chin­chilla's "decays $10\times$ over approx­i­mately $D$ tokens" [40], so its inter­me­di­ate check­points under­state a shorter run. WSD matches cosine, with "an almost per­fect match between the per­for­mance of the best cosine and cooldown sched­ule" [353]. Read the 0.38-point down­stream spread of single runs as a tie. Ending at zero is con­tested. Bergsma et al. find D2Z "con­sis­tently out­per­forms other sched­ules when train­ing at com­pute-opti­mal dataset sizes" [339], while Chin­chilla and Hägele et al. saw small costs. Our read­ing, not a mea­sured result: the edge of decay­ing to zero grows with tokens per param­e­ter and needs a higher, re-tuned peak. Modern runs train far past 20 tokens per param­e­ter, so end at 3% of peak or below and check down­stream scores.

Why does the end matter? In Wen et al.'s river valley, a high rate keeps the weights oscil­lat­ing on steep hill­sides while they drift down the river on the floor, and then "the rapidly drop­ping learn­ing rate min­i­mizes the iter­ate’s oscil­la­tions, moving it closer to the river and reveal­ing true opti­miza­tion progress" [357]. Aver­ag­ing adds a second reason. Unroll the cou­pled update with $u_t$ the Adam direc­tion:

The bracket is each update's weight in the final model. Early updates should be for­got­ten and late, noisy ones aver­aged, and Bergsma et al. argue D2Z bal­ances the two [339]. At ${\eta }_{max}=3\times {10}^{-4}$, $\lambda =0.1$ and OLMo 2 7B's 953,674 steps, the last 2.4% of steps hold half the update weight under a con­stant rate, 19.1% under cosine to 10% and 22.0% under D2Z, a window 9 times longer (com­puted).

Figure 24 shows this recency toy model: the sched­ule, and below it each step's weight in the final model with the half-weight tail shaded. Switch from con­stant to cosine to D2Z and watch the tail widen, or load a pub­lished recipe. On the right are Bergsma et al.'s losses: decay-to-zero curves sit lowest, best at twice the peak rate of the $10\times$ curves.

Prac­tice has moved toward a low end (Table 5.7) and toward WSD, because a con­stant phase can be cut any­where: one run plus cheap cooldowns traces a whole loss-versus-tokens curve. Hägele et al. put the Chin­chilla suite at $2.36\times {10}^{23}$ FLOPs this way instead of $5.59\times {10}^{23}$ [353], 58% less (com­puted). Chap­ter 6 uses this for scal­ing sweeps, and mid­train­ing branches from the stable check­point (Chap­ter 7) [358].

Your defaults: warm up lin­early over 1,000 to 2,000 steps for runs of 100K steps or more, or $N$ tokens for small scal­ing runs. If you know the run length, decay lin­early to zero or cosine to a small floor and re-tune the peak. Oth­er­wise use WSD with a 10 to 20% cooldown at about half the best cosine peak. Common fail­ures: extrap­o­lat­ing a cosine past its hori­zon, com­par­ing sched­ules of dif­fer­ent lengths mid-run, and tuning on short runs.

A WSD run aver­ages 0.90 of peak learn­ing rate, a cosine-to-10 % run 0.55, yet at equal com­pute both reach about the same loss. Why does area not win?

Area buys progress along the valley floor, but at high learn­ing rate bounc­ing across the steep walls inflates the mea­sured loss, and the cooldown removes the bounce. Final loss needs progress plus a long enough low-rate phase (10⁠–⁠20% of steps), not area alone [353, 357, 359].

5.3 Batch size and the critical batch

How large should the global batch be, why do big runs ramp it up, and what must happen to the learn­ing rate and ${\beta }_2$ when it changes? A batch gra­di­ent is an aver­age, so dou­bling the batch halves its vari­ance. While noise dom­i­nates, one step on a dou­bled batch does almost the work of two. Once the aver­age is close to the true gra­di­ent, a bigger batch buys little and burns data. The cross­over is the crit­i­cal batch, and it moves during a run.

The gra­di­ent noise scale

McCandlish et al. expand the loss to second order, with true gra­di­ent $G$, Hes­sian $H$ and per-exam­ple covari­ance $\Sigma$, and find the best step for a batch of $B$ exam­ples [360]:

The simple noise scale drops the Hes­sian: total per-exam­ple gra­di­ent vari­ance over the squared gra­di­ent norm. At 0.1, 1 and 10 times the noise scale the best step is 0.09, 0.5 and 0.91 of ${\eta }_{max}$ (com­puted). Below the noise scale the step grows about lin­early with the batch. Above it, extra exam­ples are wasted.

Aver­aged over a run, this gives a hyper­bola between opti­mizer steps $S$ and train­ing tokens $D$ to reach a target loss [360, 25]:

At $B_{\text{crit}}$ a run takes $2S_{min}$ steps and $2D_{min}$ tokens [25]. At $4B_{\text{crit}}$ it takes $1.25S_{min}$ and $5D_{min}$: 2.5 times the data for 1.6 times fewer steps (com­puted). Zhang et al. instead use $B^{*}$, the batch that costs 20% more data than the min­i­mum [361]. Set­ting $D=1.2D_{min}$ gives $B^{*}=B_{\text{crit}}/5$ (com­puted), so check which name a paper means.

Mea­sur­ing it is nearly free: com­pare squared gra­di­ent norms on one device and after the all-reduce [360]:

Then ${ℬ}_{\text{simple}}\approx 𝒮/{\left|𝒢\right|}^2$. Aver­age both esti­mates over many steps before divid­ing. GPT-3's team used the noise scale "to guide our choice of batch size" [36].

What sets the crit­i­cal batch

The crit­i­cal batch "typ­i­cally increases by an order of mag­ni­tude or more over the course of train­ing" [360]. Kaplan et al. tied it to the loss [25]. Recent fits tie it to the data: "CBS scales pri­mar­ily with data size rather than model size" [361], and Power Lines and Step Law agree [341, 362]. Table 5.8 com­pares four def­i­ni­tions. Laws for the opti­mal batch size and learn­ing rate (DeepSeek LLM, Step Law) are in Sec­tion 6.6.

Figure 25 puts this cal­cu­la­tion under your con­trol. On the left, watch the data curve stay flat below $B_{\text{crit}}/5$ and climb past $B_{\text{crit}}$. On the right, public runs sit near or below the 20%-over­head line, and dashed stretches are extrap­o­la­tion.

Ramps and rescal­ing

Because the crit­i­cal batch grows, many runs grow the batch (Table 5.9). Llama 3 explains why: "We use a lower batch size early in train­ing to improve train­ing sta­bil­ity, and increase it sub­se­quently to improve effi­ciency." [6]

A new batch needs new opti­mizer set­tings. SGD's rule is linear [366], valid only far below ${ℬ}_{\text{noise}}$. For Adam, Mal­ladi et al. derive a square-root rule valid at small learn­ing rates [367]:

Out­side that regime the evi­dence splits (Table 5.10). Hold­ing ${\beta }_2$'s half-life fixed in tokens gives ${\beta }_2^{\prime }={\beta }_2^{B^{\prime }/B}$.

You train at 4 times the crit­i­cal batch. What does that cost in data and save in steps?

With $S=S_{min}(1+B_{\text{crit}}/B)$ and $D=D_{min}(1+B/B_{\text{crit}})$, the crit­i­cal batch needs $2S_{min}$ and $2D_{min}$, while $4B_{\text{crit}}$ needs $1.25S_{min}$ and $5D_{min}$: 2.5 times the data for 1.6 times fewer steps (com­puted) [360]. Pay that only when serial time binds.

Prac­ti­cal defaults. Stay at or below the 20%-over­head batch unless serial time mat­ters more than data. Ramp if early train­ing is unsta­ble. Re-tune the rate when­ever the batch changes [371]. If gra­di­ent accu­mu­la­tion changes the batch (Sec­tion 11.2), step-counted sched­ules change mean­ing.

5.4 Hyperparameter transfer: μP, depth and initialization

If you tune a 40M-param­e­ter model, can you reuse its learn­ing rate for a 7B model with the same shape but 16 times the width? Under the stan­dard param­e­ter­i­za­tion (SP), no. Yang et al.'s Tensor Pro­grams V (TP5) found that "the opti­mal learn­ing rate shifts by roughly an order of mag­ni­tude as the width increases from 256 to 8192" [372]. Llama 3's peak rates, $3\times {10}^{-4}$, $1.5\times {10}^{-4}$ and $8\times {10}^{-5}$ at widths 4,096 to 16,384 [6], fall as width to the power $-\ln 3.75/\ln 4=-0.95$ (com­puted), though depth changes too.

Why one learn­ing rate cannot fit every width

Take a hidden layer $h=Wx$ with $n\times n$ weights of vari­ance $1/n$, so $h_i$ is of order one. Adam makes every update entry about $\eta$ in size. The gra­di­ent is an outer prod­uct with $x$, so the update is aligned with $x$ and its effect sums coher­ently:

So hidden rates must scale as $1/n$. The embed­ding's one-hot input moves $h$ by about $\eta$ at any width, so its rate must not shrink. The read­out needs $1/n$ plus a smaller output mul­ti­plier. One global rate fails all three: in SP the logits "blow up with width", and a lower rate means "the word embed­dings are not learned in large width models" [372].

Theory says a param­e­ter­i­za­tion either learns fea­tures or fol­lows kernel ("lazy") dynam­ics in which fea­tures barely move, "but not both" [373, 374, 375]. The Max­i­mal Update Param­e­ter­i­za­tion (μP) keeps every layer's fea­tures moving. Equiv­a­lently, scale spec­tral norms of weights and updates as $(\text{fan-out}/\text{fan-in}{)}^{1/2}$, with the embed­ding as the excep­tion [376]:

What μP changes

In prac­tice μP equals SP at a base width of 256 and scales with $m_{\text{width}}=d/d_{\text{base}}$ [377]:

Atten­tion uses $1/d_h$ instead of $d_h^{-1/2}$ because trained queries and keys cor­re­late [372]. At Llama 3's fixed $d_h=128$ that is just a con­stant factor of 0.088 (com­puted). Table 5.11 lists the rules.

Run­ning and test­ing μTrans­fer

Tune a proxy at base width with the target's depth, tok­enizer and data, then check an inter­me­di­ate width. TP5 sug­gests a min­i­mum "width (e.g. 256), depth (e.g., 4), batch size (e.g., 32)" and more [372], and the proxy batch should reach its crit­i­cal batch [377]. Learn­ing rate, ini­tial­iza­tion and mul­ti­pli­ers trans­fer, but reg­u­lar­iza­tion does not [372]. First run a coor­di­nate check: a few widths, about ten steps, acti­va­tion sizes flat in width [378, 379]. Table 5.13 col­lects public evi­dence.

Figure 26 shows trans­fer in data. On the left, Lingle's min­i­mum sits at $2^{-6}$ from 2M to 10B param­e­ters, though at 10B $2^{-4}$ nearly ties (2.169 against 2.167, reported) [381]. On the right, a toy MLP trained for this guide: under SP the best rate halves per width dou­bling, from $2^{-7}$ at width 128 to $2^{-10}$ at 1,024, while under μP it stays at $2^{-7}$ or $2^{-8}$ (illus­tra­tive).

What μP does and does not buy

Read together, the cri­tiques in Table 5.14 sug­gest that most of μP's gain comes from a few per-layer choices, not the full recipe. Kosson et al.'s argu­ment is short. Under AdamW's cou­pled decay, with unit-size update entries roughly orthog­o­nal to a $K\times C$ matrix $W$, the weights settle where shrink­age bal­ances growth [385]:

A sim­u­la­tion at $K=C=256$, $\eta =3\times {10}^{-3}$, $\lambda =0.1$ set­tles at 31.5 against 31.35 pre­dicted (illus­tra­tive). In equi­lib­rium only $\eta \lambda$ mat­ters, so μP's $\eta /m_{\text{width}}$ shrinks wide layers' updates unless decay is inde­pen­dent of the rate. Equi­lib­rium arrives within $1/(\eta \lambda )$ steps: 33,333 for OLMo 2 7B [7], 3.5% of its 953,674 (com­puted).

Depth, epsilon and ini­tial­iza­tion

A pre-LN resid­ual stream sums $2L$ branches, so its scale grows with depth unless each branch is damped:

Depth-μP and Bor­de­lon et al. use $\alpha =1/2$ [388, 389]. CompleteP uses $\alpha =1$, trans­fers from 2 to 128 layers and reports 11.8% FLOP sav­ings at 1.5B [390]. The expo­nent is con­tested, so keep the proxy near the target depth.

Adam's $ϵ$ belongs to the param­e­ter­i­za­tion too: it breaks the sym­me­try between mul­ti­plier, ini­tial­iza­tion and rate [383], and bites hard­est on wide models (Sec­tion 5.1). Adam-atan2 removes it [386].

Vari­ance prop­a­ga­tion sets ini­tial­iza­tion:

He ini­tial­iza­tion uses a stan­dard devi­a­tion of $(2/n_{\text{in}}{)}^{1/2}$ for ReLU layers [391]. The common 0.02 is LeCun at width 2,500: 0.55 times LeCun at width 768 but 2.56 times at 16,384 (com­puted). GPT-2 scales resid­ual-layer weights by $n_{\text{res}}^{-1/2}$ for $n_{\text{res}}$ resid­ual layers (the paper writes $N$) [19], Mega­tron-LM by $(2L{)}^{-1/2}$ for $L$ layers [392], which keeps the resid­ual vari­ance flat in depth:

Here $a$ is the atten­tion output, and ${\sigma }_{\text{out}}=0.02(2L{)}^{-1/2}$ makes the sum inde­pen­dent of $L$. OLMo 2 instead uses a flat 0.02 for every param­e­ter, which in an abla­tion cut the gra­di­ent spike score from 0.40 to 0.03 at slightly slower con­ver­gence [7]. DeepSeek-V3 uses 0.006 [2].

Prac­ti­cal defaults. Use μP-style per-layer rates with inde­pen­dent weight decay, or in SP at least a width-inde­pen­dent embed­ding rate [387, 386]. "If < 10 trials in a study, only tune the (base) learn­ing rate." [393] Lower the trans­ferred rate for a longer run [394]. Pre-LN and the other norm choices are in Sec­tion 4.3.

Under μP with Adam, hidden matri­ces' learn­ing rate is divided by the width mul­ti­plier but the embed­ding's is not. Why?

A one-hot input means a step of $\eta$ per embed­ding entry changes the output by about $\eta$ at any width. A hidden matrix's update lines up with its $n$-dimen­sional input and adds up to about $\eta n$, which a $1/n$ rate can­cels. Divid­ing the embed­ding's rate too would freeze it at large width [372, 387].

5.5 Keeping a large run stable

Why do large runs spike or diverge, and what does each stan­dard fix do mechan­i­cally? Three fail­ures hide under "insta­bil­ity", and each needs its own fix. A fast spike is a loss jump within a few steps: PaLM 540B spiked "roughly 20 times during train­ing, despite the fact that gra­di­ent clip­ping was enabled" [34]. A slow diver­gence is a quan­tity, such as the atten­tion logits, drift­ing up for thou­sands of steps until loss blows up [271]. A silent degra­da­tion never spikes: updates are lost to round­ing (Sec­tion 5.6) [182].

Warn­ing signs usu­ally come first. In GLM-130B "a train­ing col­lapse usu­ally lags behind a “spike” in gra­di­ent norm by a few train­ing steps" [395]. OLMo 2 mea­sures spik­i­ness with a spike score, the per­cent­age of points at least seven stan­dard devi­a­tions from the rolling mean of the pre­vi­ous 1,000 [7]:

A syn­thetic trace with 12 spikes in 20,000 steps scores 12 / 19,000 = 0.0632% (com­puted). What to mon­i­tor during a run is in Sec­tion 12.5.

These fail­ures can be stud­ied cheaply: they "also appear in small models when train­ing at high learn­ing rates". Worts­man et al. score a fix by its LR sen­si­tiv­ity, the mean gap to the best loss ${ℒ}^{*}$ across a learn­ing-rate sweep, where $𝒜(\eta )$ is the model trained at $\eta$ and diverged runs count as the ini­tial loss ${ℒ}_0$ [270]:

Atten­tion logits and QK-norm

The logit $z_{ij}=⟨q_i,k_j⟩/\surd d_h$, with $q=W_{Q}x$ and $k=W_{K}x$, is qua­dratic in the weights: double both matri­ces and it grows four­fold. As weights grow, the soft­max turns one-hot, and ViT-22B diverged this way near 8B param­e­ters [278]. In Worts­man et al.'s sweeps "all points with atten­tion logits above 1e4 diverged", and a logit forced to about ${10}^3$ already degraded the loss [270].

QK-norm [279] nor­mal­izes each head's queries and keys before the dot prod­uct. With RMSNorm and gains $g_q,g_k$, the Cauchy-Schwarz inequal­ity bounds every logit:

Growth must now go through a few gain param­e­ters. QK-norm let a 1.2B model train at learn­ing rate 0.3 [270], and Gemma 3 adopted it in place of soft-cap­ping [265]. Soft-cap­ping, $c\tanh (z/c)$, starves large logits of gra­di­ent: at $c=30$ its slope ${sech}^2(z/c)$ is 0.005 at $z=100$ (com­puted). Multi-head latent atten­tion cannot use QK-norm, so Kimi K2 caps them with QK-Clip (Sec­tion 5.7) [72].

Output logits and z-loss

Soft­max is shift-invari­ant, so cross-entropy cannot see a common drift of all output logits $y$: its gra­di­ent $p-e_t$, with $e_t$ the one-hot target, sums to zero. z-loss adds the miss­ing force by penal­iz­ing the log of the nor­mal­izer $Z$:

Output-logit diver­gence "occurs in models with no weight decay regard­less of scale, and z-loss resolves this insta­bil­ity". Weight decay also mit­i­gates it in larger models [270]. In BF16, whose spac­ing at 128 is 1.0 (com­puted), "A round­off error of 0.5 in bfloat16 will alter the soft­max output by 36% and incor­rectly make all logits equal." [314].

Adam, embed­dings and data

Adam can start a spike itself: in CLIP train­ing, spikes "con­sis­tently occur 1⁠–⁠8 iter­a­tions after the squared gra­di­ents become under-esti­mated by their AdamW second moment esti­ma­tor" [396]. A long-quiet coor­di­nate has a tiny $v_t$, so one large gra­di­ent becomes a huge step. A lower ${\beta }_2$ short­ens that memory (Sec­tion 5.1). Gra­di­ents below $ϵ$ can also freeze, then flip into an update "essen­tially uncor­re­lated with the direc­tion of descent" [342].

Small embed­dings are a second source, since the next norm ampli­fies their gra­di­ents. For RMSNorm with­out its gain (or LayerNorm on a cen­tered input), the Jaco­bian is

With the common ini­tial­iza­tion $𝒩(0,2/(5d))$ at width 4,096, the stan­dard devi­a­tion is 0.0099 and the scale about 101 (com­puted). Scal­ing embed­dings toward unit stan­dard devi­a­tion removes it [397].

Data is the trig­ger, not the whole cause. Replay­ing PaLM's spike batches from an ear­lier check­point gave no spike, so "spikes only occur due to the com­bi­na­tion of spe­cific data batches with a par­tic­u­lar model param­e­ter state" [34]. OLMo 2 fil­ters the repeated n-grams its spike batches were rich in (Sec­tion 3.2) [7]. Table 5.15 col­lects the fixes, and norm place­ment is in Sec­tion 4.3.

Recov­ery

The clas­sic cure is PaLM's: restart "from a check­point roughly 100 steps before the spike started" and skip "roughly 200⁠–⁠500 data batches". PaLM was "fully bit­wise repro­ducible from any check­point" [34]. Whether skip­ping cures any­thing is con­tested. It "only works well ear­lier in train­ing" [342], and in small models it "merely post­pones sub­se­quent spikes" [398]. Resum­ing from an aver­age of recent check­points may help [400]. Llama 3 chose pre­ven­tion [6]. The run­book is in Sec­tion 12.6.

Figure 27 steps through one spike on an illus­tra­tive trace. Watch the max atten­tion logit cross ${10}^3$ first, the gra­di­ent norm spike before the loss, and the rewind branch spike again because the drift was never fixed. The last step adds QK-norm.

Layer 0's max atten­tion logit has grown from 50 to 8,000 over 20K steps and the loss wob­bles. What is hap­pen­ing, and what is the fix?

Logit growth: a logit scales with the prod­uct of the query and key weight norms, and every run above ${10}^4$ diverged in Worts­man et al.'s sweeps [270]. QK-norm bounds each logit by $d_h^{1/2}$ times two gains, about 11 at $d_h=128$ (com­puted, Sec­tion 5.5) [279]. With MLA, use QK-Clip [72].

5.6 Numeric precision: BF16, FP8 and below

Which number for­mats do LLMs train in, why do we still keep FP32 copies of some things, and what does it take to train in FP8 or FP4? A float trades range, counted in binades (powers of two), against pre­ci­sion, set by its $M$ man­tissa bits. Bit lay­outs, ranges and block scal­ing are in Sec­tion 2.3. BF16 keeps FP32's range with 7 man­tissa bits [52, 314]. FP16 has 10 man­tissa bits but flushes mag­ni­tudes below $2^{-24}$ to zero [51]. FP8 E4M3 (4 expo­nent, 3 man­tissa bits) tops out at 448, E5M2 at 57,344 [117]. How train­ing and serv­ing pre­ci­sion enter the scal­ing law is in Sec­tion 6.6.

Mixed pre­ci­sion and loss scal­ing

The stan­dard recipe is "single-pre­ci­sion master weights and updates, loss-scal­ing, and accu­mu­lat­ing FP16 prod­ucts into FP32." Loss scal­ing mul­ti­plies the loss by $S$ so that small FP16 gra­di­ents do not under­flow [51]:

BF16 removed the need: "The need for loss scal­ing can be avoided by using BFLOAT16 datatype." [52]. BLOOM's 104B float16 trials diverged, and bfloat16 "proved to solve the insta­bil­ity prob­lem" [401].

Why FP32 master weights: swamp­ing

BF16 has range to spare but few man­tissa bits. Adding $\Delta w$ to $w$ with round-to-near­est loses the update when it is below half an ulp (unit in the last place) of $w$:

For FP16 the thresh­old starts at 2,048, the paper's "at least 2048 times larger" [51], and for BF16 at 256 (com­puted). With Gopher's BF16 param­e­ters "many param­e­ters did not reg­is­ter updates over many steps", and "Stor­ing a float32 copy of the param­e­ters in the opti­miser state alle­vi­ates this issue." [182]. Sto­chas­tic round­ing is unbi­ased [123]:

Yet Gopher found that "sto­chas­tic round­ing does not fully recover mixed pre­ci­sion train­ing per­for­mance" [182].

Figure 28 lets you run this. On the left, pick a format, learn­ing rate and weight, and watch round-to-near­est stall while FP32 climbs. On the right, as out­liers grow from ${10}^4$ to ${10}^5$ times typ­i­cal, one FP8 scale per tensor flushes ordi­nary values to zero while finer scales do not.

Why FP32 accu­mu­la­tion

Long sums swamp too, and float­ing-point addi­tion is not asso­cia­tive [402]. Llama 3 accu­mu­lates gra­di­ents over micro-batches in FP32 and reduce-scat­ters them in FP32 (Sec­tion 11.2). With its image encoder, "per­form­ing gra­di­ent accu­mu­la­tion in bf16 led to numer­i­cal insta­bil­i­ties" [6].

FP8: one scale per what?

FP8 matrix mul­ti­plies (GEMMs) run at twice the BF16 rate on an H100, 1,979 against 989.5 TFLOP/s dense (reported) [4]. The orig­i­nal advice is E4M3 for weights and acti­va­tions and E5M2 for gra­di­ents [117]. E4M3's roughly 18 binades cannot hold every tensor at one scale, so each block gets its own, set by its amax (largest abso­lute value):

A sim­u­la­tion in this guide (illus­tra­tive) shows why: a 1,024 × 4,096 Gauss­ian acti­va­tion with 8 out­lier chan­nels scaled by $r$. Per-tensor E4M3 holds a 2.65% error up to $r={10}^3$, but at ${10}^5$ it flushes 29.8% of ordi­nary values to zero, against 3.4% for 1×128 tiles. Real models get there, with a few acti­va­tions "100,000 times larger" than the rest [404]. With local scales, E4M3 can serve every tensor.

Table 5.17 col­lects other public runs. Fail­ures arrive late, so short val­i­da­tions mis­lead. Speedups are bounded: if a frac­tion $f$ of step time is GEMMs that run twice as fast,

which gives 1.43× at $f=0.6$ and 1.82× at 0.9 (com­puted). The reported 1.34× to 1.43× sit inside.

FP4, as of Novem­ber 2026

FP4 train­ing is research. The NVFP4 run above kept about 15% of linear layers in higher pre­ci­sion, used sto­chas­tic round­ing only for gra­di­ents, and found that BF16 for the decay "mit­i­gates the loss gap" [121]. Whether Hada­mard trans­forms rescue FP4 gra­di­ents is dis­puted (Table 5.17). One account of why low pre­ci­sion bites late: "when the gra­di­ent norm falls below approx­i­mately $\surd 3$ times the quan­ti­za­tion noise, quan­tized train­ing becomes less effec­tive" [122]. That fits both late gaps (our read­ing).

Pre­ci­sion and deter­min­ism

Reduc­tion order changes the low bits even at infer­ence: under BF16 and greedy decod­ing, a 7B rea­son­ing model's accu­racy varied by "up to 9%" across GPU counts, GPU types and eval­u­a­tion batch sizes [402]. Train­ing runs inherit the same non-asso­cia­tiv­ity. A gap "within the accept­able range of train­ing ran­dom­ness" [2] means some­thing only once that ran­dom­ness is mea­sured (Sec­tion 12.7).

5.7 New optimizers, honest benchmarks and weight averaging

Many new opti­miz­ers claim speedups of about 1.2⁠–⁠2× over AdamW. Which claims sur­vive care­ful com­par­i­son, how do the sur­vivors work, and what does weight aver­ag­ing add? Matrix opti­miz­ers led by Muon sur­vive with a modest edge that shrinks with scale. Aver­ag­ing check­points gives much of a learn­ing-rate decay for free.

An opti­mizer is a norm

Pick a norm and a sharp­ness esti­mate ${\lambda }_{max}$. The step min­i­miz­ing $g^{\top }\Delta w+\frac{{\lambda }_{max}}{2}\Vert \Delta w{\Vert }^2$ takes its direc­tion from the norm and its size from the dual norm $\Vert ·{\Vert }^{†}$:

Bern­stein and New­house show that "after switch­ing off expo­nen­tial moving aver­ages, each method is equiv­a­lent to steep­est descent under a par­tic­u­lar norm" [408]. Adam becomes sign descent. Sham­poo becomes steep­est descent under the spec­tral norm, step­ping along $U{V}^{\top }$ for $G=U\Sigma V^{\top }$: same sin­gu­lar vec­tors, every sin­gu­lar value set to 1. That norm bounds how much an update can change a layer's output [409].

Sham­poo and SOAP. For an $n_1\times n_2$ matrix ($n_1\le n_2$), Sham­poo pre­con­di­tions from both sides [410]:

For one Llama 3 8B MLP matrix (4,096 × 14,336) the fac­tors hold $({4096}^2+{14336}^2)/(4096\times 14336)=3.79$ times its entries. For the 32 layers' weight matri­ces (embed­ding and head excluded), unblocked FP32 fac­tors and inverse roots take about 183.5 GiB, against 52.0 GiB for AdamW's two FP32 moments on the same 6.98B param­e­ters (com­puted). A 128,256-row embed­ding alone would add a 128,256 × 128,256 factor, $1.6\times {10}^{10}$ entries, which is why imple­men­ta­tions block such ten­sors or leave them on AdamW. Blocked and grafted, Sham­poo costs "at most a 10% per­for­mance reduc­tion in per-step wall-clock time" [411]. SOAP runs Adam in Sham­poo's eigen­ba­sis and reports, in the large-batch regime, over 40% fewer iter­a­tions than AdamW at 360M-660M param­e­ters (reported) [412].

Muon: orthog­o­nal­ized momen­tum

Muon "opti­mizes 2D neural net­work param­e­ters by taking the updates gen­er­ated by SGD-momen­tum, and then apply­ing a Newton-Schulz (NS) iter­a­tion as a post-pro­cess­ing step". Embed­dings, the output layer and vec­tors stay on AdamW [413]. Momen­tum updates "are almost low-rank matri­ces" [413], so orthog­o­nal­iz­ing gives rare direc­tions the same step as dom­i­nant ones. Moon­light and Kimi K2 use [409, 72]

Scale $X_0=M_t/\Vert M_t{\Vert }_F$ so the sin­gu­lar values lie in $[0,1]$. Each Newton-Schulz step

keeps the sin­gu­lar vec­tors and maps each sin­gu­lar value through $\phi (x)=ax+b{x}^3+c{x}^5$, with no SVD. Jordan et al. use "the coef­fi­cients $(3.4445,-4.7750,2.0315)$" for five steps [413]. They map nor­mal­ized sin­gu­lar values from 0.01 to 1 into a band of about $[0.68,1.13]$ (com­puted), and "$\epsilon$ can be as high as around 0.3 with­out harm­ing the loss curve" [413].

Five NS steps cost $5·2(2n_2{n}_1^2+n_1^3)$ FLOPs against $6n_1{n}_{2}B$ for train­ing the layer on $B$ tokens, at most $5n_1/B$. Jordan et al. give "$5*768/524288=0.7\%$" for the GPT-2 speedrun and "$5*16384/16000000=0.5\%$" at Llama 405B scale [413]. Summed over Llama 3 405B's matri­ces at 16,777,216 tokens it is 0.39% (com­puted). At 17.7M-33.6M param­e­ters with ref­er­ence ker­nels, Muon "con­sumes 56.1⁠–⁠69.6% of train­ing time" [414].

The evi­dence. Muon improved the NanoGPT speedrun record "by a factor of 1.35x" and trained a 1.5B model to GPT-2 XL's HellaSwag score in 10 8×H100-hours against AdamW's 13.3 [413, 415], 1.33× (com­puted). Moon­light added two fixes [409]. Weight decay keeps weights from "exceed­ing the high-pre­ci­sion range of bf16". RMS match­ing: a full-rank orthog­o­nal update has $\mathrm{R}\mathrm{M}\mathrm{S}(O)=max(n_1,n_2{)}^{-1/2}$, only 0.0043 for 405B's 16,384 × 53,248 MLP (com­puted), and $0.2max(n_1,n_2{)}^{1/2}$ lifts it to AdamW's 0.2, so "Muon can directly reuse the learn­ing rate and weight decay tuned for AdamW". Moon­light reports "about 52% train­ing FLOPs" of AdamW (reported) from fits ${ℒ}_{\text{Muon}}=2.506C^{-0.052}$ and ${ℒ}_{\text{AdamW}}=2.608C^{-0.054}$ [409]. For equal loss they give 0.48 at a loss of 2.5 but 0.56 at 2.0 (com­puted), and $C$ has no stated units.

Scalar alter­na­tives fare worse. Sophia claims "a 2x speed-up com­pared to Adam" [416], but on BERT and T5 its gains and Lion's "vanish com­pared to a base­line with a fully-decayed learn­ing rate" [417].

Bench­mark­ing opti­miz­ers hon­estly

"Papers often don't suf­fi­ciently tune their AdamW base­line before com­par­ing it to a newly pro­posed opti­mizer" [413]. Wen et al. tuned eleven opti­miz­ers equally at 130M-1.2B param­e­ters. Tuning the learn­ing rate alone gave "Up to a 2 $\times$ speedup" over the GPT-3 recipe, and matrix opti­miz­ers' edge fell "from 1.4 $\times$ over AdamW for 0.1B param­e­ter models to merely 1.1 $\times$ for 1.2B param­e­ter models", with Muon pro­jected slightly behind AdamW at 7B in the 1× Chin­chilla regime [418]. In Figure 29, com­pare the author-reported points, which mostly sit at 1.3⁠–⁠2×, with the inde­pen­dent line slop­ing toward 1.

A fair test tunes both opti­miz­ers equally, com­pares fully decayed runs, since rank­ings "can flip during train­ing due to learn­ing rate decay" [418], and sweeps sizes, data ratios and batch sizes [417, 419, 370].

Opti­mizer X reaches AdamW's final loss at step 6,000 of a 10,000-step cosine run. Why is that not a 1.67× speedup?

At step 6,000 AdamW has not decayed its learn­ing rate, and rank­ings can flip during the decay [418]. A fair test reruns both with 6,000-step sched­ules tuned with the same budget, com­pares final losses, and repeats at sev­eral sizes and batch sizes [417, 370].

Weight aver­ag­ing

Aver­ag­ing can­cels noise only across decor­re­lated check­points, so gains are "ampli­fied when check­points are sam­pled with con­sid­er­able spac­ing" [420]. It also acts as a decay. Write later check­points as a base ${\theta }_n$ minus the updates $g$ since. Merg­ing with weights $c_j$ sum­ming to 1 gives

"equiv­a­lent to apply­ing a syn­thetic decay sched­ule" [421]. A uni­form aver­age of 9 check­points scales the 8 updates by 0.889, 0.778, down to 0.111, a linear decay (com­puted). Llama 3 aver­ages check­points "during anneal­ing", DeepSeek-V3 keeps a weight EMA, and OLMo 2 aver­ages sev­eral anneals (Sec­tion 7.3) [6, 2, 7]. Merg­ing con­stant-phase check­points lifted a 1.3B-active MoE's HumanEval "from 31.1 to 36.6 points" [400], yet LAWA (latest-weight aver­ag­ing, a run­ning aver­age of recent check­points) scored below the AlgoPerf base­line [356]. Whether aver­ag­ing replaces decay is open.

5.8 Reading the loss curve, and a starting recipe

What does a healthy loss curve look like, which odd shapes are normal, and what set­tings should a new run start from? Oscil­la­tion, an early bump and plateaus are usu­ally expected, so know them before you kill a run (Sec­tion 15.2).

The edge of sta­bil­ity. On a qua­dratic with top Hes­sian eigen­value ${\lambda }_{max}(H)$, iter­a­tion diverges once

the last for Adam's pre­con­di­tioned sharp­ness [422]. Net­works climb to the thresh­old and sit on it: "the train­ing loss behaves non-mono­ton­i­cally over short timescales, yet con­sis­tently decreases over long timescales" [423]. For Llama 3 405B it is $38/(8\times {10}^{-5})=\mathrm{475,000}$ at peak and $4.75\times {10}^7$ at the final $8\times {10}^{-7}$ (com­puted). The learn­ing rate sets how sharp a region the model can occupy, which is why the cooldown drops the loss (Sec­tion 5.2).

Bumps and plateaus. In-con­text learn­ing "devel­ops abruptly in a narrow window early in train­ing (roughly 2.5 to 5 bil­lion tokens)" as induc­tion heads form, "the only place in train­ing where the loss is not convex" [424]. At GPT-2 124M's 524,288 tokens per step that is steps 4,800⁠–⁠9,500, if it car­ried over (illus­tra­tive). Plateaus in small trans­form­ers come with rep­e­ti­tion and col­lapsed hidden states, also early in Pythia and OLMo [425]. Grokking (val­i­da­tion catch­ing up ${10}^3$ times later) and double descent matter for multi-epoch fine-tuning on small data [426, 427], not single-epoch pre­train­ing.

A healthy run falls fast through warmup, may bump early, declines smoothly, drops in the cooldown and ends flat, while the spike score, max atten­tion logit and $\log Z$ stay small (Sec­tion 5.5). Table 5.18 col­lects the chap­ter's defaults. Sec­tion 15.3 maps symp­toms to causes.

6. Scaling laws: predicting loss, sizing runs and spending compute

A scal­ing law makes a run plan testable before the expen­sive run starts. Fit the rela­tion between loss, model size and data, inspect the range where it holds, and decide whether train­ing com­pute or the cost of serv­ing the result is the budget that mat­ters.

Can you know how good a model will be before you pay to train it? In 2022 DeepMind bet that you could. Its 2021 Gopher had 280B param­e­ters trained on 300B tokens [182], because "most models are trained for approx­i­mately 300 bil­lion tokens" [40]. A scal­ing law fitted to small runs said the same budget belonged to a 67B model on 1.5T tokens, about 4 times smaller on 5 times more tokens (reported, their Table 3) [40]. Hoff­mann et al. built one: "a more com­pute-opti­mal 70B model, called Chin­chilla, on 1.4 tril­lion tokens". It beat Gopher on MMLU "with an aver­age accu­racy of 67.6% (improv­ing upon Gopher by 7.6%)" [40].

In 2024 Meta bet the other way. Llama 3 8B saw about 15T tokens, about 1,870 tokens per param­e­ter or 93 times Chin­chilla's ratio (com­puted: $15\times {10}^{12}/8.03\times {10}^9$), on pur­pose: "we also train our smaller models for much longer than is com­pute-opti­mal" [6]. Meta also sized its 405B flag­ship from runs of at most ${10}^{22}$ FLOPs, and its bench­mark fore­cast "only slightly under­es­ti­mates the final per­for­mance of the flag­ship Llama 3 model" [6].

A scal­ing law is an empir­i­cal fit that pre­dicts held-out loss (and, with more work, bench­mark scores) from param­e­ters $N$, train­ing tokens $D$ and com­pute $C$, for one recipe on one data dis­tri­bu­tion. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. Sec­tions 6.1 to 6.4 cover loss laws and how to fit them, 6.5 and 6.6 deci­sions beyond com­pute-opti­mal, 6.7 and 6.8 bench­marks, test-time and RL com­pute, and 6.9 the trend lines.

6.1 Power laws and the first scaling laws

Why does loss fall as a power law, and what did the first lan­guage-model scal­ing laws mea­sure? Loss falls smoothly and slowly with scale: each 10x of resources buys about the same frac­tional gain. That is the sig­na­ture of a power law $y=a{x}^{-k}$, a straight line of slope $-k$ on log-log axes. Loss cannot fall below the entropy of the text, so the work­ing form adds a floor, the irre­duc­ible loss $E$:

In Henighan et al.'s read­ing, "the irre­duc­ible loss esti­mates the entropy of the true data dis­tri­bu­tion, while the reduc­ible loss is an esti­mate of the KL diver­gence between the true and model dis­tri­bu­tions" [428]. Only the reduc­ible loss $ℒ-E$ is straight on log-log axes, so plot $\log (ℒ-\widehat{E})$ too, and treat the fitted floor as the least cer­tain param­e­ter.

The early evi­dence came fast. Hes­t­ness et al. found that the expo­nent "usu­ally set­tles between $-0.07$ and $-0.35$", small­est for lan­guage models ($-0.0656$ for a word model, reported), and that "Improved model archi­tec­tures and opti­miz­ers can improve the power-law inter­cept, but not the expo­nent" [429]. Rosen­feld et al. pro­posed addi­tive data and model terms over a floor [430]. Henighan et al. found the same form in images, video and math, with com­pute-opti­mal size grow­ing as about $C^{0.7}$ in every domain [428].

Kaplan et al. trained models from 768 to 1.5B non-embed­ding param­e­ters (token and posi­tion embed­dings excluded) on WebText2, with "a 3000 step linear warmup fol­lowed by a cosine decay to zero" [25]. Com­pute is about $6N$ FLOPs per token (Sec­tion 1.2), quoted in PF-days of $8.64\times {10}^{19}$ FLOPs. Each law below holds when that one factor limits train­ing, across "eight orders of mag­ni­tude" in com­pute [25]. The con­stants depend on the tok­enizer, and none of these forms has a floor.

Kaplan's com­pute-effi­cient fron­tier (their Table 6) put most new com­pute into size [25]:

where $C_{min}$ is com­pute adjusted to small-batch train­ing. Ten times the com­pute buys 5.4 times the model and 1.9 times the tokens (com­puted: ${10}^{0.73}$, ${10}^{0.27}$), while steps grow only as $C_{min}^{0.03}$. Hence "Big models may be more impor­tant than big data" [25].².

The sample effi­ciency of large models sur­vived. The allo­ca­tion rule did not (Sec­tions 6.2 and 6.3), and Kaplan et al. warned that their fron­tier breaks near ${10}^4$ PF-days ($8.6\times {10}^{23}$ FLOPs, com­puted) and 1.7 nats per token, "though the numer­i­cal values are highly uncer­tain" [25]. That is where Gopher and Chin­chilla were trained. Nei­ther that 1.7 nor Chin­chilla's $E=1.69$ is the entropy of Eng­lish: datasets and tok­eniz­ers differ.

Why power laws at all? No theory yet pre­dicts mea­sured LLM expo­nents (as of Novem­ber 2026). Lead­ing accounts tie the expo­nent to the dimen­sion of the data man­i­fold [431] or to a long tail of skills learned in order of fre­quency [432]. Other the­o­ries, forms that bend or break, and data prun­ing that beats the law are in [433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 289, 445]. Three mis­takes recur: fit­ting a pure power law to data with a floor, com­par­ing expo­nents across dif­fer­ent def­i­ni­tions of $N$ or $C$, and assum­ing an expo­nent is uni­ver­sal when data changes it (Sec­tion 6.3).

6.2 Compute-optimal training: the Chinchilla analysis

Given a fixed FLOP budget, how should you split it between param­e­ters and train­ing tokens? At fixed $C\approx 6ND$, a bigger model sees fewer tokens. Too small cannot rep­re­sent the data, too big stops under­trained, and in between is a min­i­mum. Plot final loss against size for runs of equal com­pute, an IsoFLOP pro­file, and you see a valley.

Hoff­mann et al. trained "over 400 lan­guage models rang­ing from 70 mil­lion to over 16 bil­lion param­e­ters on 5 to 500 bil­lion tokens" [40], count­ing total param­e­ters and full FLOPs (within a few per­cent of $6ND$ above 1B param­e­ters, 1.10 times it at 305M, reported). They esti­mated $N_{\text{opt}}\propto C^a$ and $D_{\text{opt}}\propto C^b$ three ways [40]. Approach 1 trains each size with four cosine sched­ules of dif­fer­ent lengths, then, at each of 1,500 com­pute values, takes the lowest loss any run reached there (every winner came from the last 15% of its sched­ule). Approach 2 trains many sizes at each of 9 bud­gets from $6\times {10}^{18}$ to $3\times {10}^{21}$ FLOPs, finds "a clear valley in loss", and fits a par­a­bola in $\log N$ to locate each min­i­mum. Approach 3 fits a para­met­ric loss to every run:

with $E$ the loss of an ideal model and the other terms the short­falls of finite size and finite data. Here $B$ and $\beta$ are fit con­stants, not a batch size or a KL coef­fi­cient. The fit min­i­mizes a Huber loss (squared error for small resid­u­als, abso­lute error for large ones, so out­liers count less) on log losses, using L-BFGS, a quasi-Newton opti­mizer, with thresh­old $\delta ={10}^{-3}$, and as printed is $ℒ(N,D)=1.69+406.4/N^{0.34}+410.7/D^{0.28}$ [40]. The paper's TeX source car­ries more digits for three of the five con­stants, $E=1.6934$, $\alpha =0.3392$ and $\beta =0.2849$, while $A$ and $B$ are the same [446]. A 2024 refit (Sec­tion 6.3), this chap­ter's default, gives $E=1.8172$, $A=482.01$, $B=2085.43$, $\alpha =0.3478$, $\beta =0.3658$ [446].

Hence the head­line: "for every dou­bling of model size the number of train­ing tokens should also be dou­bled" [40]. Figure 30 draws the val­leys from the fitted law. Notice how flat each one is near the bottom, and how the line through the minima moves when you switch between printed, unrounded and refit con­stants. Drag the size off the opti­mum to see what it costs.

Deriv­ing the fron­tier

Assume the law is exact, $C=6ND$, data is seen once and hyper­pa­ram­e­ters are tuned every­where. Sub­sti­tute $D=C/(6N)$:

Set­ting $dℒ/dN=0$ gives $\alpha A{N}^{-\alpha -1}=\beta B(6/C{)}^{\beta }{N}^{\beta -1}$, so

and $a+b=1$. With error terms $X=A{N}^{-\alpha }$ and $Y=B{D}^{-\beta }$, the opti­mum bal­ances $\alpha X^{\star }=\beta Y^{\star }$. The expo­nents set how the split changes with scale. The con­stants only set the ratio $D/N$. Along the fron­tier ${ℒ}^{\star }(C)-E\propto C^{-\gamma }$ with $\gamma =\alpha \beta /(\alpha +\beta )$, 0.178 under the refit and 0.155 unrounded (com­puted), the range Pearce and Song report [43].

Where 20 tokens per param­e­ter comes from

The paper never states it as a rule. Approach 1's pro­jec­tions (400M on 8.0B tokens up to 1T on 21.2T) imply 20.0 to 22.4 tokens per param­e­ter, and Chin­chilla's 70B on 1.4T is exactly 20, $6ND=5.88\times {10}^{23}$ FLOPs (com­puted) [40]. Re-analy­ses call 20 the con­sen­sus [446, 447]. With $D=20N$, $C=120N^2$, so

91B on 1.83T at ${10}^{24}$ FLOPs (com­puted). The table com­pares the fits, ending at Llama 3 405B's budget.

At Gopher's budget, Chin­chilla out­per­formed Gopher, GPT-3 and Mega­tron-Turing NLG on a large range of tasks, MMLU 67.6% [40]. It also switched to AdamW and changed the tok­enizer slightly, so this is not a pure allo­ca­tion abla­tion. A cleaner test at ${10}^{21}$ FLOPs: Approach 1 said "a 2.86 bil­lion param­e­ter model should be opti­mal" against Kaplan's 4.68B, and a 2.80B model beat a 4.74B one (reported) [40].

Approach 3 pre­dicted smaller models than the other two, which the authors blamed on cur­va­ture: "pro­jec­tions from very small models lead to dif­fer­ent pre­dic­tions than those from larger models" [40]. Most of the gap was a fit­ting bug (Sec­tion 6.3).

What a wrong size costs

Train $N=\rho N_{\text{opt}}$ at the same $C$. Then $X=X^{\star }{\rho }^{-\alpha }$ and $Y=Y^{\star }{\rho }^{\beta }$. The bal­ance con­di­tion gives $X^{\star }=\beta R^{\star }/(\alpha +\beta )$ and $Y^{\star }=\alpha R^{\star }/(\alpha +\beta )$ for the reduc­ible loss $R=X+Y$, so

The second ratio, the share of the budget a com­pute-opti­mal model would need for the same loss, fol­lows from $R^{\star }\propto C^{-\gamma }$ and does not depend on $C$. Mis-sizing wastes the same share at every scale. The deriva­tion is ours, checked against brute-force root find­ing (com­puted).

Fail­ure modes: the printed Approach 3 con­stants imply about 93 tokens per param­e­ter at Chin­chilla's budget (com­puted), so keep them out of fron­tier deci­sions. Com­pute-opti­mal ignores serv­ing cost (Sec­tion 6.5) and assumes unique data (Sec­tion 6.6). The loss values are MassiveText nats and do not trans­fer.

You train a model 2x larger than com­pute-opti­mal at a fixed budget. Roughly what frac­tion of the com­pute did you waste, and does the answer depend on the budget?

About 16% under the Besiroglu et al. refit [446]. At $\rho =N/N_{\text{opt}}=2$ the reduc­ible loss grows by $(\beta {\rho }^{-\alpha }+\alpha {\rho }^{\beta })/(\alpha +\beta )=1.031$, and a com­pute-opti­mal model reaches that loss with ${1.031}^{-1/\gamma }\approx 0.84$ of the com­pute, $\gamma =\alpha \beta /(\alpha +\beta )=0.178$ (com­puted). Both the fron­tier and the penalty are power laws in $C$, so the frac­tion is the same at every budget. At 4x it is about 49%.

6.3 Why Kaplan and Chinchilla disagreed

Two care­ful stud­ies dis­agreed on the most impor­tant number in LLM plan­ning, 0.73 versus 0.5. What went wrong, and what does the answer teach about mea­sure­ment? At Gopher's 6,667 PF-days, Kaplan's Table 6 asks for $1.3\times {10}^9\times {6667}^{0.73}=804$B non-embed­ding param­e­ters on $2\times {10}^{10}\times {6667}^{0.27}=216$B tokens (com­puted) [25], 11 times Chin­chilla. The two fron­tiers cross near ${10}^{19}$ FLOPs (com­puted, approx­i­mate), about where Kaplan mea­sured, and diverge by an order of mag­ni­tude at fron­tier bud­gets. Four expla­na­tions fol­lowed.

Sched­ules. Kaplan used one long cosine, so check­points of long runs had not decayed, and Chin­chilla found that over­shoot­ing the cosine "by more than 25%" hurts [40]. This proved minor. Porian et al. find that "care­ful learn­ing rate decay is not essen­tial for the valid­ity of their scal­ing law", and Pearce and Song that train­ing one model per size, with­out anneal­ing, moves the fitted coef­fi­cient only marginally [27, 43]. Sched­ules are com­pared in Sec­tion 5.2.

Embed­dings at small scale. Pearce and Song argue that "much of this dis­crep­ancy can be attrib­uted to Kaplan count­ing non-embed­ding rather than total param­e­ters, com­bined with their anal­y­sis being per­formed at small scale" [43]. Non-embed­ding param­e­ters grow as $d^3$, embed­dings as $d$, and their fit to Chin­chilla's con­fig­u­ra­tions is

so embed­dings are half the model near ${\omega }^{3/2}\approx {10}^7$ param­e­ters (com­puted). In $N_{⧵E}$, the opti­mal size is no longer a power law. Its local expo­nent has two limits,

0.76 and 0.51 under the refit, and over­shoots both in between, to about 0.97 near ${10}^6$ param­e­ters (com­puted). Over Kaplan's sizes their sim­u­la­tion gives "0.74 to 0.78 (close to Kaplan's 0.73)", and their own tiny models gave 0.49 with total counts and 0.74 with non-embed­ding counts [43].

Head FLOPs, warmup and tuning. Porian et al. repro­duced Kaplan's expo­nent with 16 models from 5M to 901M param­e­ters and "over 900 train­ing runs", then removed one dif­fer­ence at a time, "iden­ti­fy­ing three fac­tors caus­ing the dif­fer­ence: last layer com­pu­ta­tional cost, warmup dura­tion, and scale-depen­dent opti­mizer tuning" [27]. Figure 31 steps through the cor­rec­tions. Watch the expo­nent fall and the range of com­pute-opti­mal tokens per param­e­ter shrink to about 15. Its table view lists every inter­val and $D/N$ range (reported, Porian et al. Table 1).

Leav­ing out the head under-counts com­pute "from roughly 10% at larger models to roughly 90% at smaller models", so small models look cheap and the slope tilts toward big ones. Kaplan's 3,000 warmup steps of $2^{19}$ tokens are 1.57B tokens (com­puted), so small models "are ‘forced’ to use more train­ing tokens than would oth­er­wise be opti­mal" [27]. A fixed batch leaves small models only a few hun­dred steps, and tuning per size closes the rest (Sec­tion 5.3). The final fit matches 0.5 within 0.6% and Chin­chilla's size within 15% [27]. Porian et al. released the 900+ runs and code. Kaplan's study was care­ful for 2020, and its account­ing choices only matter at small scale, which is where lad­ders run.

Data qual­ity. DeepSeek LLM got $a$ = 0.450 on its early data, 0.524 on its cur­rent data and 0.578 on OpenWebText2, Kaplan's own dataset (reported): "The higher the data qual­ity, the more the increased com­pute budget should be allo­cated to model scal­ing" [42]. That spread is smaller than the 0.23 gap, and with DeepSeek's account­ing Kaplan's own data gives 0.58, not 0.73. DeepSeek also replaced $6N$, off by up to 50% at small scale, with non-embed­ding FLOPs per token $M=72n_{\text{layers}}{d}^2+12n_{\text{layers}}ds$ (the paper's $n_{\text{layer}},d_{\text{model}},l_{\text{seq}}$), and fit $M_{\text{opt}}=0.1715C^{0.5243}$, $D_{\text{opt}}=5.8316C^{0.4757}$ [42], coef­fi­cients whose prod­uct is 1.0001 (com­puted).

So Kaplan's 0.73 was mostly an arti­fact of small-scale account­ing (embed­dings and head, which Kaplan tied) and of a warmup and batch that hand­i­capped small models. Decay is minor, and data qual­ity moves the expo­nent by a few hun­dredths. Count total param­e­ters and all FLOPs, tune per scale, and fit a floor.

Under the refit, tokens per param­e­ter falls slowly with scale (18.4 at $5.76\times {10}^{23}$, 16.1 at ${10}^{26}$) because $a>0.5$. Under the unrounded con­stants it rises (59 at $5.76\times {10}^{23}$, 93 at ${10}^{26}$, com­puted). The sign of $\beta -\alpha$ decides, and the data cannot reli­ably tell $a=0.49$ from 0.51 (Sec­tion 6.4).

In Porian et al.'s repro­duc­tion of Kaplan, which single change moved the fitted expo­nent the most, and why does leav­ing it out push the expo­nent up?

Count­ing the output layer's FLOPs: 0.835 to 0.706 on RefinedWeb [27]. With­out it, small models' com­pute is under-counted the most, so the best small-budget runs look cheaper than they are and the fitted fron­tier grows model size too fast with com­pute.

6.4 How to fit a scaling law you can trust

You have budget for a ladder of small runs and must choose the size of a big one. How do you design the ladder, fit the law, and know how far to trust the extrap­o­la­tion? Sec­tion 6.3 showed that a few setup choices moved the expo­nent from Kaplan's 0.73 to about 0.5 [25, 27], so most of the work is keep­ing the small runs honest. Table 6.4 lists the pro­ce­dure in seven steps. Four carry most of the risk.

Design the ladder off the diag­o­nal. If every run sits on one ray $D=kN$, the $N$ and $D$ terms of the law cannot be told apart, and designs off that ray win 97.3% of held-out com­par­isons [448]. Cere­bras-GPT's family, all near 20 tokens per param­e­ter, is a fine set of models and a weak dataset for fit­ting five con­stants [377]. A con­stant learn­ing rate with short cooldowns branched from check­points "saves half the time and FLOPs": Chin­chilla's suite would cost $2.36\times {10}^{23}$ FLOPs instead of $5.59\times {10}^{23}$ (reported esti­mate) [353, 354].

Fit two ways. The IsoFLOP route fits a par­a­bola in $\log N$ at each budget, then a power law through the minima [40, 6, 27]. The para­met­ric route fits all five con­stants with Chin­chilla's objec­tive [40]:

Here $A=e^a$, $B=e^b$, $E=e^e$ and $\delta ={10}^{-3}$, so $a$, $b$ and $e$ are log-con­stants, not the fron­tier expo­nents. The log-sum-exp keeps the con­stants pos­i­tive, and the small Huber thresh­old makes this nearly an abso­lute-error fit that resists out­liers. Sum the terms, since aver­ag­ing them stopped Chin­chilla's L-BFGS early (Sec­tion 6.3) [446]. Start from a grid of ini­tial values and check con­ver­gence. Even the train­ing opti­mizer changes the expo­nents [453]. If the two routes agree, the func­tional form is prob­a­bly ade­quate.

Put error bars on the deci­sion. Boot­strap over runs, not over cor­re­lated check­points of one run, and refit each resam­ple to con­ver­gence. Besiroglu et al.'s boot­strap makes their "80% con­fi­dence inter­val is roughly 50-fold wider than that reported by Hoff­mann et al." [446]. Many near-equal fits extrap­o­late dif­fer­ently, because "stan­dard scal­ing law fit­ting is sub­ject to a Rashomon effect" [452].

Extrap­o­late with care. Chin­chilla's own small and large models gave dif­fer­ent pro­jec­tions (Sec­tion 6.2) [40]. The expo­nent is the dan­ger­ous number. With $N_{\text{opt}}=G{C}^a$ and the fit piv­ot­ing at a budget $C_p$ where its error is zero,

A real fit pivots near the middle of its ladder. For Llama 3, $C_p\approx (6\times {10}^{18}\times {10}^{22}{)}^{1/2}=2.4\times {10}^{20}$, so an expo­nent error of 0.005 moves the pre­dic­tion at $3.8\times {10}^{25}$ by $e^{0.005\times 12.0}=1.06$ times (com­puted). Keep a printed pref­ac­tor while round­ing the expo­nent, and the pivot is $C_p=1$ FLOP: the lever is $\ln C=58.9$ and the same 0.005 is worth 1.34 times (com­puted). Llama 3 extrap­o­lated $3.8\times {10}^{25}/{10}^{22}=\mathrm{3,800}$ times past its largest budget (com­puted).

Newer tools choose which ladder runs to train [454] and refine the func­tional form [455]. As of Novem­ber 2026 they are promis­ing, not stan­dard.

6.5 Beyond compute-optimal: overtraining and inference-aware sizing

If a model will serve bil­lions of requests, how big should it be and how long should you train it? Smaller and longer than Chin­chilla says. Train­ing costs about $6ND$ FLOPs once. Serv­ing costs about $2N$ FLOPs per token for the model's whole life (Sec­tion 1.6). A smaller model is cheaper on every served token but needs more train­ing tokens to reach the same loss. Past some serv­ing volume, the sav­ings repay the extra train­ing. Chin­chilla counts only the first term.

The infer­ence-aware objec­tive

Sar­dana et al. fix a target loss ${ℒ}^{\star }$ and min­i­mize life­time com­pute, with $D_{\text{inf}}$ the tokens served [70]:

The con­straint gives the train­ing tokens for each size:

The search is over $N$ alone. A model of 30B-Chin­chilla qual­ity that will serve ${10}^{13}$ tokens should be a 13.6B model on 2.84 times the data, saving 28% of life­time FLOPs (reported). With Sar­dana et al.'s con­stants ($E=1.69$, $A=406.4$, $B=410.7$, and $\alpha =0.336$, $\beta =0.283$, which they take from De Vries to fit Hoff­mann et al.'s Table A3 better than the rounded 0.34 and 0.28) our repro­duc­tion matches [70]. Chin­chilla's unrounded con­stants give 13.9B, 2.74 times and 26.5%, the Besiroglu refit 11.6B, 4.25 times and 43% (com­puted) [40, 446]. The direc­tion is robust. The size depends on the con­stants, so name them. Dollar results shift fur­ther, since serv­ing uti­liza­tion can be about 1% against 40⁠–⁠60% in train­ing [70], but we could not repro­duce the paper's dollar fig­ures, so we do not use them.

Figure 32 runs the opti­miza­tion live. Raise the serv­ing volume and the filled marker (life­time opti­mum) slides left of the hollow one (com­pute-opti­mal model of equal qual­ity). The "Llama 3 8B-like" preset, 30B qual­ity and ${10}^{14}$ served tokens, lands on 7.75B param­e­ters and 16.2T tokens, about 2,090 per param­e­ter (com­puted, Sar­dana et al.'s con­stants).

Over­train­ing is pre­dictable

Qual­ity keeps improv­ing far out: "model qual­ity con­tin­ues to improve as we scale tokens per param­e­ter to extreme ranges (up to 10,000)", but con­stants fitted near 20:1 "over­es­ti­mate the impact of addi­tional train­ing data" there, and a refit on all runs drops $\alpha$ and $\beta$ from 0.34 and 0.28 to 0.18 and 0.24 [70]. Gadre et al. pre­dicted the loss of a 1.4B model on 900B tokens, 32 times over­trained, within 0.7% from runs using 300 times less com­pute (reported), though "Scal­ing can break down when under-train­ing." [456] Their trick: assume $\alpha =\beta$ and sub­sti­tute $N=(C/(6D/N){)}^{1/2}$ and $D=(C(D/N)/6{)}^{1/2}$:

Here $a^{″}=A{6}^{\alpha /2}$ and $b^{″}=B{6}^{\alpha /2}$. At fixed tokens per param­e­ter, reduc­ible loss is a straight line in $\log C$ with slope $-\alpha /2$, and chang­ing $D/N$ shifts the line with­out tilt­ing it, which is why small over­trained runs pre­dict large ones. The bracket is small­est at $(D/N{)}_{\text{opt}}=(B/A{)}^{1/\alpha }$. Write $D/N=\kappa (D/N{)}_{\text{opt}}$, with $\kappa$ the over­train­ing factor, and the bracket becomes $2(a^{″}{b}^{″}{)}^{1/2}\cosh (\frac{\alpha }{2}\ln \kappa )$, so the com­pute needed for the com­pute-opti­mal loss grows by (derived here)

What real models do

Pre-Chin­chilla models sat at 1 to 2 tokens per param­e­ter, Chin­chilla moved the field to about 20, and served models now sit at hun­dreds to thou­sands (Table 6.5): "Llama 3 8B is over­trained by close to 100x, while Llama 3 70B is only over­trained by 10x" [457].

Figure 33 puts these models against the com­pute-opti­mal lines of Kaplan, Chin­chilla and the Besiroglu refit. Nearly every served model sits in the over­trained region above the Chin­chilla line, while the flag­ships (Chin­chilla, Llama 3 405B, DeepSeek-V3 per total param­e­ter) hug it. For a mix­ture of experts, tokens per active param­e­ter is the ratio that com­pares with dense com­pute.

Con­se­quences and defaults

At fixed com­pute $D=(C(D/N)/6{)}^{1/2}$, so over­train­ing by $\kappa$ needs ${\kappa }^{1/2}$ times more tokens (derived) [457], which brings the data wall closer (Sec­tion 6.6). Over­trained models lose more to post-train­ing quan­ti­za­tion [458], and same-size models differ by up to 3.5 times in latency with shape [459]. To do it your­self, pick the deploy­ment tier, fore­cast life­time traf­fic, then train as long as data and budget allow, keep­ing a flag­ship near com­pute-opti­mal [70, 456]. The clas­sic mis­take is fore­cast­ing a 2,000:1 run with con­stants fitted near 20:1.

6.6 Repeated data, mixture of experts, precision, vocabulary and hyperparameters

What do scal­ing laws say when unique data runs short, when the model is sparse, when you train in fewer bits, when the vocab­u­lary grows, and when you must pick a learn­ing rate for a run you cannot sweep? Each exten­sion adds one axis to the Chin­chilla law, and each con­clu­sion can be an arti­fact of what it held fixed. Table 6.7 col­lects them.

Repeated data and the data wall

What repeated data is worth is derived in Sec­tion 3.6. The head­line is that up to 4 epochs of repeated data cost almost noth­ing against unique data [209], while a 2026 law with an over­fit­ting term finds that "Beyond a point, fur­ther rep­e­ti­tion is coun­ter­pro­duc­tive" [460]. Vil­lalo­bos et al. esti­mate about 510T tokens on the indexed web and expect train­ing sets to match the public human text stock "between 2026 and 2032, or slightly ear­lier if models are over­trained" (median 2028) [457]. Sec­tion 6.9 returns to that limit.

Mix­ture of experts

Mix­ture-of-experts (MoE) laws flip with what they hold fixed (archi­tec­ture in Sec­tion 4.7).

At fixed com­pute "the opti­mal spar­sity level increases with model size", though sparser models reason worse at equal per­plex­ity [312]. More experts favor fewer active param­e­ters and more tokens, and that study's dense opti­mum at ${10}^{22}$ FLOPs is 4.7 tokens per param­e­ter (com­puted), so 20:1 is not uni­ver­sal [462]. Other laws cover expert counts and sparse net­works [463, 464, 465]. DeepSeek-V3's $6\times 37\times {10}^9\times 14.8\times {10}^{12}=3.3\times {10}^{24}$ FLOPs is 11.6 times less than Llama 3 405B's (com­puted) [2], before rout­ing traf­fic (Sec­tion 13.6).

Pre­ci­sion

Kumar et al. fit over 465 runs and treat low-pre­ci­sion train­ing as a smaller effec­tive param­e­ter count [458]:

with $P$ the bits for weights, acti­va­tions and KV cache and each $\gamma$ fitted. They report an opti­mal train­ing pre­ci­sion of about 7⁠–⁠8 bits for inte­gers at every budget (reported). With the pub­lished con­stants, 4 bits every­where leaves $N_{\text{eff}}/N\approx 0.19$ (com­puted), in line with their warn­ing that "the race to low-pre­ci­sion train­ing may have to stop before going below 4-bits" [458]. Fitted con­stants alone move the opti­mum by about two bits (com­puted).³. With­out off­sets and with one tied con­stant $\overline{\gamma }$, max­i­miz­ing $(1-e^{-x}{)}^3/x$ with $x=P/\overline{\gamma }$ gives $e^{x^{\star }}=1+3x^{\star }$, so $x^{\star }\approx 1.904$ and $P^{\star }=x^{\star }\overline{\gamma }=5.1$ bits for $\overline{\gamma }=2.6745$ (com­puted). Serv­ing adds a twist: "the degra­da­tion intro­duced by post-train­ing quan­ti­za­tion increases as models are trained on more data, even­tu­ally making addi­tional pre­train­ing data actively harm­ful" [458]. A float­ing-point law puts the best pre­ci­sion at 4⁠–⁠8 bits [466], other laws cover quan­ti­za­tion-aware train­ing and low-bit infer­ence [467, 468, 469], and DeepSeek-V3 trained in FP8 [2] (mechan­ics in Sec­tion 2.3, Sec­tion 5.6 and Sec­tion 13.8).

Vocab­u­lary

Vocab­u­lary param­e­ters should grow more slowly than the rest, and Tao et al. pre­dict that Llama 2 70B should have had a vocab­u­lary of at least 216K, 7 times its 32K [249]. Down­stream evi­dence is thin: ARC-Chal­lenge rose from 29.1 to 32.0, about two of the reported stan­dard devi­a­tions (1.3 to 1.4 points), on one task [249, 251]. Tok­eniz­ers are in Sec­tion 3.8.

Learn­ing rate and batch size

Nobody sweeps a ${10}^{25}$-FLOP run, so labs fit hyper­pa­ram­e­ter laws on small ones. DeepSeek LLM fit them in com­pute $C=MD$, with $M$ the non-embed­ding FLOPs per token, and found a broad basin [42]:

Step Law, fit on over 3,700 models and 100T tokens, sets the batch from $D$ alone [362]. Bjorck et al. find that at fixed batch "the opti­mal LR decreases as the token hori­zon gets longer", as $D^{-0.32}$, so a 10 times longer run wants 0.48 times the learn­ing rate (com­puted) [394]. The crit­i­cal batch size is derived in Sec­tion 5.3. It "scales pri­mar­ily with data size rather than model size" [361, 360], and Zhang et al.'s fit for a 302M model, $B_{\text{crit}}=22.91D^{0.47}$, extrap­o­lates to tens of mil­lions of tokens at 15T (com­puted, far out­side the fitted range) [361, 341] against Llama 3 405B's final 16M [6].

The laws dis­agree by about 40 times on the 8B (Table 6.6) because each depends on what its sweep held fixed. Bjorck fixes the batch, so a longer run means more steps at the same noise. Step Law grows the batch as $D^{0.571}$, which under Adam's square-root rule (Sec­tion 5.3) raises the learn­ing rate as $D^{0.29}$, close to its $D^{0.307}$ (our inter­pre­ta­tion) [362, 386, 286]. Fit your own at three or more scales, scale the batch with data, and cut the learn­ing rate roughly as $D^{-0.3}$ when you lengthen a run at fixed batch.

Sched­ules and dis­til­la­tion

Cosine, WSD and con­stant-with-cooldown sched­ules are com­pared in Sec­tion 5.2. Tissue et al.'s anneal­ing law is derived in Sec­tion 7.1. Sched­ules shift every con­stant: with warmup-stable-decay, MiniCPM found "the data size should be 192 times larger than the model size on aver­age, as opposed to 20 times in Hoff­mann et al. (2022)" [354]. The cost and scal­ing law of dis­til­la­tion are in Sec­tion 7.7, which explains Gemma 2's small models trained on over 50 times their com­pute-opti­mal tokens [268]. Par­al­lel infer­ence has a pro­posed law too [470].

Three pub­lished laws give Llama 3 8B learn­ing rates of about $3\times {10}^{-4}$, $2\times {10}^{-3}$ and $4\times {10}^{-5}$. Which one is right?

None in gen­eral. Each is con­di­tional on what its sweep held fixed. At a fixed batch size a longer run means more steps at the same gra­di­ent noise, which wants a smaller learn­ing rate. A law that grows the batch with data allows a larger one. A law in com­pute alone ignores the split between $N$ and $D$. Sched­ule and param­e­ter­i­za­tion shift all three, so fit your own.

6.7 From loss to capabilities: downstream prediction and emergence

Loss falls smoothly, but nobody ships a loss. Can you pre­dict a bench­mark score before train­ing, and do some abil­i­ties appear sud­denly? Often yes, with wide error bars, and many claimed jumps trace back to the metric. Loss aver­ages bil­lions of tokens, so it moves smoothly. A bench­mark is a few thou­sand thresh­olded pass/fail events, which turn a smooth input into a noisy S-curve. So every work­ing method pre­dicts a smooth inter­me­di­ate, then maps it to the score with a bounded link.

Two-step fore­casts

OpenAI pre­dicted GPT-4's final loss from models trained with "at most 10,000x less com­pute", and its HumanEval mean log pass rate, a loss-like quan­tity, from models with at most 1,000x less. It held on five of six dif­fi­culty buck­ets [472]. Llama 3 made the two steps explicit: fit the neg­a­tive log-like­li­hood of the cor­rect answer against com­pute on the scal­ing-law ladder, then fit a sig­moid from log-like­li­hood to accu­racy, with Llama 2 models sup­ply­ing points high on the curve [6]. On ARC-Chal­lenge the fore­cast came in slightly low, with no error number given [6]. From the largest ladder budget, ${10}^{22}$ FLOPs, to the flag­ship's $3.8\times {10}^{25}$ is ${\log }_{10}\mathrm{3,800}=3.6$ decades (com­puted).

The OLMo 2 ladder pub­lishes the recipe and a score­card. Bhagia et al. train four sizes (190M to 1.3B non-embed­ding param­e­ters), each on 1, 2, 5 and 10 times 20 tokens per param­e­ter: 16 runs at about 1% of the tar­gets' com­pute. Task loss is bits per byte of the cor­rect answer. Step 1 fits a Chin­chilla-form law per task on $(N,D)$, because "FLOPs cannot dis­tin­guish between com­pute-opti­mal and over­trained models." Step 2 fits, on about 1,400 check­points [473],

where $b^{\prime }$ is chance, $a^{\prime }+b^{\prime }$ the ceil­ing, ${ℒ}_0$ the mid­point and $k^{\prime }$ (neg­a­tive once fitted) the steep­ness. Table 6.8 shows the score­card.

Tasks with the largest misses also varied most across ladder check­points, a free warn­ing sign [473]. Report fore­casts per task, with error bars.

Aggre­gates are easier than single tasks. Gadre et al. pre­dict the aver­age error over 17 tasks within 1 point for a 6.9B model from runs cost­ing 20x less, but "indi­vid­ual down­stream task pre­dic­tions are sig­nif­i­cantly more noisy" [456]. Revis­it­ing 46 tasks, Lourie et al. find only 18 (39%) improve smoothly and pre­dictably [474], and mul­ti­ple-choice accu­racy is hard to fore­cast because it com­pares the cor­rect choice "against a small number of spe­cific incor­rect choices" [475].

Emer­gence

Wei et al. call an abil­ity emer­gent "if it is not present in smaller models but is present in larger models." GPT-3's few-shot scores on sev­eral tasks jumped from chance at about $2\times {10}^{22}$ train­ing FLOPs. On six such BIG-Bench tasks, LaMDA's cross-entropy kept improv­ing at small scale while exact match stayed flat [476]. Scha­ef­fer et al. argue that "non­lin­ear or dis­con­tin­u­ous met­rics pro­duce appar­ent emer­gent abil­i­ties", and that over 92% of claimed BIG-Bench emer­gent abil­i­ties appear under mul­ti­ple-choice grade or exact string match [477]. Their toy model shows why. Let per-token cross-entropy fall as ${ℒ}_{\text{tok}}(N)=(N/c{)}^{-\theta }$, so the right token has prob­a­bil­ity $q_{\text{tok}}=e^{-{ℒ}_{\text{tok}}}$. With inde­pen­dent tokens, exact match on an $\ell$-token answer is

With illus­tra­tive $c={10}^5$ and $\theta =0.25$, going from ${10}^7$ to ${10}^{12}$ param­e­ters lifts per-token accu­racy from 0.73 to 0.98 and 10-token exact match from 0.04 to 0.84 (com­puted). Figure 34 lets you change the answer length and switch the y-axis to a log scale. The jump on the linear axis van­ishes on the log axis. The shaded band marks scores below $1/n$, which $n$ test items cannot resolve.

Not every jump is the metric. Plot­ting against pre­train­ing loss, Du et al. find models "per­form at the random level until the pre-train­ing loss decreases to about 2.2", a break that sur­vives con­tin­u­ous met­rics [478]. If $ℒ(N)=E+(N_0/N{)}^{{\alpha }_N}$ at fixed data, a thresh­old ${ℒ}_{\text{th}}$ is crossed at

where the floor $E$ now includes the data term. With the printed Chin­chilla law [40] and an illus­tra­tive thresh­old of 2.0 nats, that is about 13B param­e­ters at 1.4T tokens but 3.7B at 15T (com­puted). More data moves "emer­gence" to smaller models. (Du et al.'s 2.2 belongs to their corpus and tok­enizer, so never com­pare it with another loss.)

A model's per-token accu­racy rises from 0.95 to 0.97 on a task whose answers are 20 tokens long. What hap­pens to exact match, and is this emer­gence?

With inde­pen­dent tokens, exact match goes from ${0.95}^{20}=0.36$ to ${0.97}^{20}=0.54$ (com­puted), up by half from a 2-point per-token change. That is metric non­lin­ear­ity, not a new abil­ity. The log-prob­a­bil­ity of the full answer, $20\ln q_{\text{tok}}$, moves smoothly from $-1.03$ to $-0.61$.

6.8 Scaling test-time compute and RL compute

Once a model is trained, how does per­for­mance scale with com­pute spent per prob­lem, and how does it scale with rein­force­ment-learn­ing com­pute? Test-time com­pute has two axes: par­al­lel (sample many answers, pick one) and sequen­tial (think longer). Sam­pling fol­lows a power law set by prob­lem dif­fi­culty. RL reward fol­lows a sig­moid because rewards are bounded.

Repeated sam­pling and the heavy tail

Cov­er­age is the frac­tion of prob­lems solved by at least one of $k$ sam­ples, which is pass@k under a per­fect ver­i­fier. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct's cov­er­age "increases from 15.9% with one sample to 56% with 250 sam­ples", and Brown et al. model cov­er­age as an expo­nen­ti­ated power law, $\log c(k)\approx -u{k}^{-v}$ [479]. The unbi­ased pass@k esti­ma­tor is derived in Sec­tion 10.3.

Why a power law? One prob­lem with per-sample suc­cess prob­a­bil­ity $q$ has $\text{pass@}k=1-(1-q{)}^k$, which sat­u­rates expo­nen­tially. If every prob­lem had $q=0.159$, pass@250 would be $1-{0.841}^{250}\approx 1.0$, not 56% (com­puted), so many prob­lems must have $q$ near zero. Let $q$ have den­sity $g(q)$ across prob­lems. For large $k$ only prob­lems with $q\lesssim 1/k$ still fail, and if $g(q)\propto q^{\lambda -1}$ near zero,

The expo­nent is the shape of the hard tail: "per-prob­lem expo­nen­tial scal­ing can be made con­sis­tent with aggre­gate poly­no­mial scal­ing if the dis­tri­bu­tion of single-attempt suc­cess prob­a­bil­i­ties is heavy tailed" [480]. For $q~\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}(\lambda ,\mu )$ it is exact:

With $\mathrm{B}\mathrm{e}\mathrm{t}\mathrm{a}(0.3,3)$, cov­er­age is 0.09, 0.37, 0.67, 0.83 and 0.92 at $k=1,10,100,{10}^3,{10}^4$, and the log-log slope of $1-c$ at large $k$ is $-0.30=-\lambda$ (com­puted). The tail can be miss­ing. On Best-of-N jail­break­ing prompts, Llama 3 8B Instruct has no heavy tail of near-zero suc­cess rates and so no aggre­gate power law, although its suc­cess on each prompt still rises expo­nen­tially with attempts [480]. The "A pool of tasks" mode of Figure 53 (Sec­tion 10.3) plots cov­er­age up to $k=\mathrm{10,000}$ beside Brown et al.'s points. The slow climb after $k=100$ is the tail.

Cov­er­age is only an upper bound. On MATH with Llama 3 8B Instruct, cov­er­age rose from 82.9% at 100 sam­ples to 98.44% at 10,000, but major­ity voting or reward-model selec­tion went only from 40.50% to 41.41% [479]. Voting accu­racy con­verges to a limit set by the model's output dis­tri­bu­tion [481]. The ver­i­fier sets what you real­ize (Sec­tion 14.6).

Test-time com­pute against a bigger model

Snell et al. find that choos­ing the test-time strat­egy per ques­tion by its esti­mated dif­fi­culty (best-of-N, beam search against a ver­i­fier, or sequen­tial revi­sions) matches best-of-N with up to 4x less test-time com­pute [71]. They also ask when a small model with extra test-time com­pute beats one $g$ times larger. Pre­train­ing costs $6N{D}_{\text{pre}}$ FLOPs and serv­ing $2N{D}_{\text{inf}}$, both $g$ times more for the larger model, so the small model can mul­ti­ply its infer­ence com­pute by $K$ before total spend matches: $6N{D}_{\text{pre}}+2N{D}_{\text{inf}}K=g(6N{D}_{\text{pre}}+2N{D}_{\text{inf}})$, so

Empir­i­cally, test-time com­pute wins on easy ques­tions or at low infer­ence load, and the larger model wins on hard ques­tions or at high load [71]. This is Sec­tion 6.5's infer­ence-aware argu­ment from the other side. The $2N$ per token in that for­mula under­counts long traces: by Brown et al.'s per-token for­mula [479], atten­tion is 12% of Llama 3 8B's FLOPs at 4K con­text and 81% at 128K (com­puted).

Think­ing longer, and RL com­pute

DeepSeek-R1-Zero's AIME 2024 pass@1 rose from 15.6% to 77.9% during RL, and its think­ing time grew with­out any exter­nal push [3]. s1 extends think­ing by append­ing "Wait", with gains that flat­ten at about six exten­sions [482]. GRPO is derived in Sec­tion 9.3.

RL com­pute is still small next to pre­train­ing. R1-Zero's RL used about 101K H800 GPU-hours [3], or $101/\mathrm{2,664}=3.8\%$ of DeepSeek-V3's pre­train­ing GPU-hours (com­puted) [2]. ScaleRL, a study of more than 400,000 GPU-hours, fits held-out reward against RL com­pute $C_{\text{RL}}$ (GPU-hours) with a sig­moid [483]:

$R_0$ is the start­ing reward, $R_{max}$ the ceil­ing (the paper's $A$), $\omega$ the steep­ness (the paper's $B$), and $C_{\text{mid}}$ the com­pute at half the gain. The useful find­ing is a split between knobs: "loss aggre­ga­tion, nor­mal­iza­tion, cur­ricu­lum, and off-policy algo­rithm pri­mar­ily mod­u­late com­pute effi­ciency with­out mate­ri­ally shift­ing the asymp­tote", while batch size, gen­er­a­tion length, model size and FP32 pre­ci­sion at the LM head (0.52 to 0.61) move the ceil­ing. Repeat runs put $\pm 0.02$ of noise on the ceil­ing [483].

Base models over­take RL-trained models on pass@k as $k$ grows [484], yet pro­longed RL widens pass@128 where the base model starts weak [485]. Whether RL expands the rea­son­ing bound­ary is argued in Sec­tion 9.7.

RL recipe A reaches a higher reward than recipe B at 3k GPU-hours. Should you pick A for a 100k GPU-hour run?

Not on that evi­dence. Fit the sig­moid to both and com­pare ceil­ings and mid­points. In Table 6.9 the batch of 2,048 prompts has only 2% of its gain at 1k GPU-hours but 98% at 100k, with a ceil­ing of 0.645 against 0.610: it loses early and wins late. Ceil­ing dif­fer­ences under about 0.02 are run-to-run noise [483].

6.9 Trend lines and limits

What has actu­ally been scal­ing, how fast, and what could stop it by 2030? The growth of fron­tier train­ing com­pute is charted in Sec­tion 1.8: about 4⁠–⁠5x per year (Epoch AI, to May 2024) [101], or 5.5x in this guide's fit to the Novem­ber 2026 data. The com­pute needed for a fixed lan­guage-model per­for­mance halved about every 8 months (95% CI 5 to 14) [102]. A halv­ing time of $T$ months is a yearly factor of $2^{12/T}$, or 2.83x for 8 months. Mul­ti­plied by 4⁠–⁠5x per year of phys­i­cal com­pute, effec­tive com­pute grows about 11⁠–⁠14x per year, 7x to 21x across the inter­val at 4x (com­puted, assum­ing inde­pen­dence).

For capa­bil­ity, METR's time hori­zons are in Sec­tion 10.7: a 207-day dou­bling (reported) and 187.8 days on the Sep­tem­ber 2026 data (com­puted). The 50% hori­zon $H_{50}$ is the human task length that an agent com­pletes half the time [486].

What could stop it? Epoch's August 2024 anal­y­sis finds "2e29 FLOP train­ing runs will likely be fea­si­ble by 2030", and the con­straint "likely to bind first is power, fol­lowed by the capac­ity to man­u­fac­ture enough chips" [103] (Table 6.10).

Llama 3.1 405B's clus­ter needed about 27 MW. An on-trend $2\times {10}^{29}$ run is about 5,000x the com­pute at 24x better power effi­ciency, so $\mathrm{5,000}/24\times 27$ MW $=5.6$ GW (com­puted), which Epoch puts at 6 GW [103]. That is the trend, not a stretch: $3.8\times {10}^{25}\times 4^6=1.6\times {10}^{29}$ (com­puted). Data is close behind: the indexed web's roughly 500 tril­lion dedu­pli­cated tokens [103] feed, at 20 tokens per param­e­ter, a 25T-param­e­ter model and $7.5\times {10}^{28}$ FLOPs (com­puted). Train­ing sets should match the stock of public human text "between 2026 and 2032" [457], hence the inter­est in repeated and syn­thetic data (Sec­tion 6.6).

7. Midtraining: annealing, long context and continued pretraining

The last train­ing tokens arrive at a check­point that already knows a great deal. Their mix­ture, con­text length and learn­ing-rate decay can shift the model sub­stan­tially. Treat each con­tin­u­a­tion as an exper­i­ment with its own data, com­pute and eval­u­a­tion budget.

What hap­pens between pre­train­ing and post-train­ing, and why did it get its own name? Start with one result. OLMo 2 7B scored 24.1 on GSM8K, a set of grade-school math word prob­lems [488], at the end of pre­train­ing and 67.5 after mid­train­ing, which aver­aged three 50B-token runs on the Dolmino mix with dif­fer­ent data orders [7]. Those 150B tokens are 3.7% of its 4.05T train­ing tokens (com­puted: 150 / 4,050).

OLMo 2 defines mid-train­ing as a shorter second stage, 5⁠–⁠10% of train­ing FLOPs, "where we lin­early decay the learn­ing rate to zero over the remain­ing length of the run", a term it took from the Phi-4 report and OpenAI [7, 196]. Sur­veys use it more broadly [489, 490]. This guide writes it as one word. Three kinds of stage share the name (Table 7.1). For a fixed model, $C\approx 6ND$ makes a stage's token share equal its FLOP share:

Long-con­text stages break the equal­ity because atten­tion cost grows with sequence length (Sec­tion 7.4).

The chap­ter also covers train­ing that starts from a check­point: con­tin­ued pre­train­ing and dis­til­la­tion. Llama 3.2 1B and 3B were pruned in one shot from Llama 3.1 8B, then pre­trained with the logits of Llama 3.1 8B and 70B as token-level tar­gets [494].

Why a name in 2024? Our read­ing: warmup-stable-decay (WSD) sched­ules made branch­ing a decay cheap [354, 353], late upsam­pling paid off at 7⁠–⁠8B scale [6, 208], and syn­thetic rea­son­ing data became plen­ti­ful. Sec­tions 7.1–7.3 cover the decay phase, 7.4–7.5 long con­text and 7.6–7.7 check­point reuse. Fine-tuning and RL follow in Chap­ter 8 and Chap­ter 9. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

7.1 Why the last tokens count most: the decay phase

Why does loss fall when the learn­ing rate decays, and why does data seen during the decay count more than data seen ear­lier? Cosine, WSD and con­stant-with-cooldown sched­ules are com­pared in Sec­tion 5.2. At a con­stant learn­ing rate (LR), sto­chas­tic gra­di­ent descent (SGD) never reaches the bottom of the basin. It bounces at a noise floor that scales with the LR. Decay­ing the LR lets it settle, and where it set­tles depends on the data it sees mean­while.

The noise floor on one direc­tion

Take one direc­tion of a qua­dratic loss $ℒ(x)=\frac{1}{2}h{x}^2$ with cur­va­ture $h$, add inde­pen­dent gra­di­ent noise ${\xi }_t$ of vari­ance ${\sigma }^2$, and run SGD with­out momen­tum (a toy model):

At equi­lib­rium the vari­ance $v$ sat­is­fies $v=(1-\eta h{)}^{2}v+{\eta }^2{\sigma }^2$, so

The excess loss is pro­por­tional to $\eta$ and, to first order, inde­pen­dent of cur­va­ture. A direc­tion relaxes in about $1/(\eta h)$ steps, so flat direc­tions settle slowly. With $h=\sigma =1$ the for­mula gives 0.0556, 0.0263 and 0.0128 at $\eta$ = 0.2, 0.1 and 0.05, match­ing a sim­u­la­tion (com­puted). Halve the LR, halve the floor. Wen et al. pic­ture the LLM case as a river valley, where the decay removes the oscil­la­tion across the walls [357], and in a 0.2B WSD run from the MiniCPM team the cur­va­ture rose by an order of mag­ni­tude during the decay [354].

Why late data writes the model

Now let each step's data pull toward its own target $x_t^{\star }$, with $x_{t+1}=x_t-{\eta }_{t}h(x_t-x_t^{\star })$ and no noise. Unrolling gives

Every later step shrinks a target's weight, so early data fades. (Sec­tion 5.2 builds a dif­fer­ent toy from AdamW's weight decay. This one tracks cur­va­ture.) For a direc­tion that relaxes in 1,000 steps at peak LR, in a 20,000-step run, the last 10% of steps write 86.5% of the final iter­ate under con­stant LR, 63.2% under WSD with a 10% linear decay, 19.3% under cosine to 10% of peak and 1.6% under cosine to zero (illus­tra­tive toy, com­puted). Con­stant LR is more recency-heavy still but never leaves its floor. WSD lowers the floor while late data still dom­i­nates.

The anneal­ing scal­ing law

Tissue et al. fit loss after step $t$ to the for­ward area $S_1$ under the LR curve and an anneal­ing area $S_2$, a run­ning sum of LR drops with momen­tum ${\lambda }_{\text{a}}$ [359]:

For a 594M-param­e­ter model at a peak LR of $2\times {10}^{-4}$ over 20K steps, with ${\lambda }_{\text{a}}=0.999$, they fit $ℒ=2.628+0.429S_1^{-0.550}-0.411S_2$ [359]. A decay raises $S_2$, but every low-LR step adds less to $S_1$, so some decay length is best. In Figure 35, drag the decay length and watch the pre­dicted loss bottom out on a flat min­i­mum, then switch to cosine and watch the late-data share col­lapse. Panel C counts the last f of the steps (20% by default), so its WSD share reads 86.5%, not the 63.2% for the last 10% above.

How long a decay, and what goes wrong

Reports differ, partly by def­i­ni­tion. MiniCPM found an expo­nen­tial decay over 10% of tokens suf­fices and 2.5% falls short [354]. Hägele et al. saw gains plateau near 20%, with a (1-sqrt) shape, ${\eta }_t={\eta }_{max}(1-[(t-t_{\text{end}}+t_{\text{decay}})/t_{\text{decay}}{]}^{1/2})$, beat­ing linear [353]. Tissue et al. call 10⁠–⁠20% typ­i­cal [359]. Feng et al.'s best qual­ity data phase covers the last 40%, a data length, not an LR one [496].

Four fail­ure modes recur. A decay can be too short. Anneal­ing to exactly zero "can hurt met­rics" while loss falls [353]. Mixes com­pared before the decay are com­pared at the noise floor. And tuning pre­train­ing LR by mid­train­ing results mis­leads: "a higher learn­ing rate does make mid-train­ing more effec­tive, but it does so by exactly the amount that the pre­train­ing is worse" [7].

A com­pet­ing view merges check­points instead of decay­ing, and WSM reports gains over WSD this way [400, 421], while Hägele et al. found aver­ag­ing falls short of a cooldown [353]. Weight aver­ag­ing is explained in Sec­tion 5.7. Our advice: WSD with a 10⁠–⁠25% linear or (1-sqrt) decay to near zero, keep pre-decay check­points to branch from, and never com­pare data mixes before decay­ing.

7.2 What goes into a midtraining mix

What data do labs put into the decay phase, how much of it, and what does it buy? Spread over a 4T-token run, 10B tokens of excel­lent math are 0.25% of every batch. Inside a 50B-token decay they are a fifth of every batch, and noth­ing comes after to over­write them (illus­tra­tive).

The OLMo 2 recipe, step by step

1. Warm up 2,000 steps to a peak LR of $3\times {10}^{-4}$, then follow a cosine planned to reach 10% of peak at 5T tokens [7].
2. Stop at 4T tokens, because "the last part of a cosine decay sched­ule can be cut off and replaced by a linear decay to zero with little loss of per­for­mance" [7].
3. Build Dolmino Mix 1124. Its 50B ver­sion is 47.2% fil­tered DCLM web text, 20.8% math, 16.6% decon­tam­i­nated FLAN instruc­tions, 7.11% Wikipedia, 5.85% aca­demic papers and 2.45% StackExchange. The 10.7B math tokens are largely syn­thetic and include the GSM8K train­ing split. Web stays near 50% in every mix, and small sources repeat two or four times [7].
4. Decay the LR lin­early to zero over 50B tokens, three times with dif­fer­ent data orders, and aver­age the three models into a soup (Sec­tion 7.3). The 13B and 32B soup three 100B runs and one 300B run.

Decay or data?

OLMo 2's Table 9 aver­age over ten eval­u­a­tions rose from 53.0 to 62.9 at 7B, 9.9 points (com­puted), although its text says 10.6. The gain shrinks to +7.0 at 32B, yet GSM8K still rises 22.6 points there (com­puted) [7]. A 50B decay on the unchanged pre­train­ing mix raised the OLMES mul­ti­ple-choice aver­age [497] from 69.6 to 74.0 and MMLU [498] from 59.8 to 61.8, while the 200-ques­tion GSM8K devel­op­ment score (GSM*) slipped from 28.5 to 27.0. Adding math to high-qual­ity web lifted GSM* to 52.0 [7], 23.5 points above web alone. With stan­dard errors of 3.1⁠–⁠3.5 points on 200 items (com­puted), the −1.5 is noise and the +23.5 is not. Part of that jump is train­ing on the test task: the math data, here and in Table 7.3, includes the GSM8K train­ing split, and GSM* is drawn from the same dis­tri­bu­tion [7, 499]. The decay buys gen­eral gains, and the data decides which skills gain. Figure 36 has three views (SmolLM2 by stage, OLMo 2 by size, decay or data): watch where the math scores jump.

Other labs follow the pat­tern (Table 7.2). Read Llama 3 care­fully: its 8B gains of 24.0% on GSM8K and 6.4% on MATH (units unstated) come from anneal­ing on those bench­marks' train­ing sets, and were neg­li­gi­ble at 405B. The pro­duc­tion anneal excludes bench­mark train­ing sets, and the 405B's cov­ered only the last 40M tokens [6], two to three steps at its 16M-token batch (com­puted).

Instruc­tion and rea­son­ing data, and readi­ness for RL

Do instruc­tion and rea­son­ing data belong in the decay? For: OLMo 3's mix with them won "on every base eval mea­sure" [12], PRISM reports +15 to +40 math points from about 27B mid­train­ing tokens [500], and early rea­son­ing data gave a 19% gain that later SFT could not fully repli­cate [501]. Against: DeepSeek LLM's 5 mil­lion instruc­tion exam­ples in the last 10% of pre­train­ing gave results "nearly iden­ti­cal" to adding them in SFT [42]. Our read­ing: for­mats are learned any­where, but rea­son­ing pat­terns placed early sur­vive post-train­ing. Gains are largest for domains far from web text, such as code and math [502].

Defaults from these recipes, not a law: a decay of 5⁠–⁠20% of tokens, about half of it the best web data, scarce domain data at 10⁠–⁠35%, small sources repeated a few times, every­thing decon­tam­i­nated.

7.3 Anneals as experiments, soups as insurance, and the contamination trap

How do you mea­sure whether a dataset belongs in the mix, should you aver­age sev­eral anneals, and how does bench­mark leak­age creep in? Change only the end of a strong model's train­ing and see whether the target bench­mark moves. Every can­di­date gets the same decay gain, so dif­fer­ences are data effects. Small models cannot do this job, since "most LLMs trained at smaller scales reg­is­ter random accu­racy on many impor­tant bench­marks such as MMLU" [208]. How to run small-scale abla­tions and when to trust them is Sec­tion 3.9.

Llama 3 anneals a 50%-trained 8B to zero over 40B tokens, 30% of them can­di­date data [6]. OLMo 2's microan­neals pair can­di­date math with equal web text, 19 runs and 130B tokens in all [7]. OLMo 3 pairs 5B can­di­date with 5B web tokens and com­pares the result with a 10B web-only microan­neal [12].

Table 7.3 teaches three lessons [7, 503]. Domain data must be present, not dom­i­nant. A few repeats help. Format mat­ters most: code answers hurt, prose rewrites help.

Win­ners can still trade off in a full mix [12], and base scores can miss post-train­ing effects, so one agen­tic recipe scores can­di­dates after a short fixed SFT run [492, 493].

Soup­ing anneals

Weight aver­ag­ing and soups are explained in Sec­tion 5.7. OLMo 2 soups anneals that share a start and differ in data order, Llama 3 aver­ages check­points during its anneal, and OLMo 3 found merg­ing helped at 32B but not 7B [7, 6, 12]. When does it help? Let $k$ ingre­di­ents devi­ate from a shared opti­mum ${\theta }^{\star }$ by vec­tors of equal size with pair­wise cor­re­la­tion $\rho$ in the Hes­sian's inner prod­uct. Expand­ing the qua­dratic loss of the mean gives $k$ diag­o­nal and $k(k-1)$ off-diag­o­nal terms, so (derived, toy model)

The con­tam­i­na­tion trap

Mid­train­ing data is curated while watch­ing bench­marks, so leak­age con­cen­trates there. Con­tam­i­na­tion, includ­ing GPT-3's fil­ter­ing bug, is in Sec­tion 10.5. OLMo 3 found con­tam­i­na­tion in stan­dard sources such as Flan, removed over 60,000 DROP train­ing exam­ples, and focused decon­tam­i­na­tion on mid­train­ing, where mem­o­riza­tion is strong­est [12]. N-gram fil­ters miss rephras­ings [196, 504]. Train­ing on bench­mark-style task data is a sep­a­rate con­found: models trained after Novem­ber 2023 beat older ones at equal com­pute by 7 MMLU and 19 GSM8K points, a gap that van­ishes after equal task fine-tuning [499]. Hold out most of a bench­mark, decon­tam­i­nate every split, and test on prob­lems newer than the data.

7.4 Why train short, and how to stretch RoPE

Why do labs pre­train at 4K-8K tokens and extend at the end, and what exactly do Posi­tion Inter­po­la­tion, NTK-aware scal­ing, YaRN and ABF change inside RoPE? Cost and data push long con­text to the end. The obsta­cle is that a model trained at 8K with rotary posi­tion embed­dings (RoPE) breaks at 32K unless its rota­tion fre­quen­cies are rescaled.

Why short first

Atten­tion FLOPs grow with sequence length $s$. Sec­tion 1.2 com­putes that a 128K-token step costs 2.88x an 8K step per token for Llama 3 8B, 2.08x for 70B and 1.60x for 405B. Meta's stated reason: "We do not train on long sequences ear­lier because the com­pute in self-atten­tion layers grows quadrat­i­cally in the sequence length." [6] In Xiong et al.'s 7B test at a fixed token budget, train­ing at 4K for 80% of the run and 32K for the rest used $2.270\times {10}^{22}$ FLOPs against $3.783\times {10}^{22}$ for 32K through­out, a 40% saving at sim­i­lar long-con­text scores except a lower Qasper F1 (25.0 against 28.6) [505]. Their count ignores causal mask­ing. With it the saving is about 27% (com­puted). Data is the second reason: long doc­u­ments are scarce, and only about 30% of SlimPajama doc­u­ments exceed 4K tokens [506].

RoPE as a set of clocks

How RoPE encodes rel­a­tive posi­tion is in Sec­tion 4.6. It rotates pair $j$ of each $d_h$-wide query and key head by the angle $m{\omega }_j$ at posi­tion $m$, so scores depend only on rel­a­tive posi­tion [299]. The useful quan­tity is the wave­length ${\lambda }_j$, the tokens per full turn [507]:

With $d_h=128$, pair 0 turns every 6.28 tokens. Pair 63 turns every $2\pi \times {\text{10,000}}^{126/128}\approx \text{54,410}$ tokens at Llama 2's base of 10,000 [505] and every 2.56M at the 500,000 that Llama 3 used from the start [6] (com­puted).

A pair whose wave­length exceeds $s_{\text{train}}$ acts like an abso­lute-posi­tion fea­ture, and past $s_{\text{train}}$ it sees angles never seen in train­ing [507]. That is 18 of 64 pairs for Llama 2 at 4,096 tokens (pairs 46⁠–⁠63) and 29 of 64 for Llama 3 8B at 8,192 (com­puted). Liu et al.'s crit­i­cal dimen­sion, 92 of 128 for Llama 2, marks the same bound­ary, so "the last 36 dimen­sions lack suf­fi­cient train­ing" [508]. Unchanged, LLaMA's per­plex­ity past its window exceeds 1,000, and over 10,000 batches of longer fine-tuning with­out touch­ing RoPE moved its effec­tive window only from 2,048 to 2,560 tokens [509].

Four rescal­ing rules

• Posi­tion Inter­po­la­tion (PI) divides every ${\omega }_j$ by $k_{\text{ext}}$. It is stable, with an error bound about 600 times below extrap­o­la­tion, and took LLaMA from 2K to 32K in 1,000 steps [509]. But the fast clocks slow too. PI degrades beyond roughly $k_{\text{ext}}=8$, and at 16x LLaMA 7B's MMLU fell from 35.7 to 25.9, against 30.0 for YaRN [507].
• NTK-aware scal­ing raises the base to ${\theta }_{\text{base}}{k}_{\text{ext}}^{d_h/(d_h-2)}$, so the fastest pair is unchanged and the slow­est slows by exactly $k_{\text{ext}}$ [507]. For Llama 2 at $k_{\text{ext}}=8$: $\text{10,000}\times 8^{128/126}\approx \text{82,685}$ (com­puted). Its dynamic vari­ant changes the rota­tion with length, so the KV cache must store keys before rota­tion [507].
• ABF (adjusted base fre­quency) raises the base and con­tin­ues pre­train­ing long. Llama 2 Long went from 10,000 to 500,000 because the old base would not attend past 4,000⁠–⁠6,000 tokens [505].
• YaRN counts each pair's turns in the trained window, $r_j=s_{\text{train}}/{\lambda }_j$. It inter­po­lates pairs below ${\alpha }_{\text{Y}}=1$ turn like PI, leaves pairs above ${\beta }_{\text{Y}}=32$ alone, blends between, and mul­ti­plies logits by $1/t$ [507]:

A Llama 2 7B trained with YaRN only on 64K chunks reached a per­plex­ity of 2.37 at 128K [507]. The Llama 3.1 rule, as imple­mented in Hug­ging Face Trans­form­ers, is YaRN-like with factor 8: pairs with wave­lengths under 2,048 tokens stay, pairs over 8,192 are divided by 8, and a ramp blends between [510]. On Llama 3 8B that leaves 29 pairs, blends 6 and divides 29 (com­puted). Figure 37 steps through the rules on Llama 2 and Llama 3 8B, plot­ting wave­length against pair index on a log axis. Watch PI lift the whole curve, NTK-aware pivot it about pair 0, and YaRN divide the pairs above the line by $k_{\text{ext}}$, blend the pairs just below it and leave the fast pairs alone.

How big a base? Men et al. require the expected advan­tage of a sim­i­lar key over a random one, pro­por­tional to ${\sum }_j\cos (m{\omega }_j)$, to stay non-neg­a­tive, which needs a base of at least $7.8\times {10}^6$ for 128K. A smaller base gives "super­fi­cial long con­text capa­bil­ity", low per­plex­ity with­out retrieval [511]. The NTK-aware for­mula gives $8.4\times {10}^6$ for Llama 3 8B at $k_{\text{ext}}=16$ (com­puted), and ProLong chose $8\times {10}^6$ for 64K [512].

Other routes (Sec­tion 4.6): ALiBi [298], layers with­out posi­tion code mixed with RoPE layers [281, 513], and LongRoPE's searched per-pair fac­tors, which reach 2,048K [514].

7.5 Long-context recipes and the gap between claimed and effective context

How did Llama 3, DeepSeek-V3 and others extend to 128K and beyond, what did it cost, and how much of the claimed window actu­ally works? The recipes share a shape: a few per­cent of tokens, lengths raised in stages, a check before each step. On hard tests, the usable window is a frac­tion of the accepted one.

Llama 3: six stages and a stop rule

Llama 3 405B grew from 8K to 128K in six stages on about 800B tokens [6]. The reus­able part is the rule for moving on: "We assess suc­cess­ful adap­ta­tion by mea­sur­ing whether (1) model per­for­mance on short-con­text eval­u­a­tions has recov­ered com­pletely and (2) the model per­fectly solves “needle in a haystack” tasks up to that length." [6] Per-stage lengths and tokens are not given. Mask­ing atten­tion across packed doc­u­ments mat­tered at these lengths [6]. The 128K stage ran on 16,384 H100s with con­text par­al­lel­ism 16 at 380 TFLOP/s per GPU and 38% BF16 MFU, against 400 TFLOP/s and 41% at 8K [6]. Con­text par­al­lel­ism and ring atten­tion are in Sec­tion 11.5.

Long data and how much of it

Token bud­gets in Table 7.4 span three orders of mag­ni­tude. Fu et al. keep the pre­train­ing domain mix, upsam­ple long doc­u­ments per domain, and find 500M tokens unlock most retrieval [506]. Llama 2 Long kept most of its gain with most long texts removed [505]. ProLong's best mix was 60% long data (code repos­i­to­ries, books) and 40% high-qual­ity short data, with no cross-doc­u­ment atten­tion and train­ing past the eval­u­ated length [512]. Whether more tokens keep help­ing is con­tested: Qiao et al. see "no clear sign of sat­u­ra­tion" up to about 100B tokens on a 1.22B model [517]. Needle tests sat­u­rate early. Harder tasks do not.

Claimed versus effec­tive con­text

The needle-in-a-haystack test hides one sen­tence in filler at every length and depth [518]. Llama 3 finds 100% of nee­dles [6], so the test is a stop rule, not a finish line. RULER calls a length effec­tive when its 13-task aver­age beats Llama 2 7B's 85.6 at 4K. By that test Llama 3.1 8B Instruct, which claims 128K, is effec­tive to 32K and 70B Instruct to 64K, and 10 of 17 models that claim at least 32K pass there (com­puted from Table 3), which the authors sum­ma­rize as only half [519]. NoLiMa removes word over­lap between ques­tion and needle and sets the bar at 85% of each model's short-con­text score. There Llama 3.1 8B is effec­tive only to 1K, Llama 3.1 70B and 405B to 2K, and GPT-4o to 8K [520]. Models also use only 10⁠–⁠20% of their con­text on BABILong [521] and lose accu­racy when the answer sits mid-con­text [522].

Figure 38 joins each model's claimed window to its effec­tive length (results as pub­lished in 2024 and 2025). Toggle from RULER to NoLiMa and watch the filled dots slide left.

HELMET finds needle tests do not pre­dict down­stream tasks [523]. Per­plex­ity is con­tested: useful in one study [524], unre­li­able for choos­ing recipes in ProLong [512], and fixed by LongPPL, which scores only tokens that depend on dis­tant con­text and cor­re­lates with bench­marks at -0.96 [525]. Chap­ter 10 covers the sta­tis­tics.

At serv­ing time one 128K sequence holds 16 GiB of BF16 KV cache for Llama 3 8B and 63 GiB for 405B (com­puted from the pub­lished con­fig­u­ra­tions [6]). Sec­tion 13.2 derives the KV-cache arith­metic, and FlashAttention, which keeps atten­tion memory linear in $s$ [63], is in Sec­tion 2.4.

How to do it your­self

• Pre­train at 4K-8K with a base of at least 500,000. Extend at the end with ABF or YaRN. For 128K aim for a base near ${10}^7$ (Men et al.'s bound is $7.8\times {10}^6$), or use a fre­quency rule such as YaRN or Llama 3.1's on a smaller base.
• Use about 60% long and 40% high-qual­ity short data, upsam­ple long doc­u­ments per domain, mask across doc­u­ments, and train a little past the target length.
• Advance a stage only when short eval­u­a­tions recover and a long test passes a stated thresh­old, with RULER or HELMET, not nee­dles alone.
• Report effec­tive length with bench­mark and thresh­old, and re-test after SFT and RL.

7.6 Continued pretraining for a new domain

How do you teach a trained model a large new domain or lan­guage with­out eras­ing what it already knows? Con­tin­ued pre­train­ing (CPT) is more next-token train­ing of a fin­ished model on new or reweighted data at pre­train­ing scale. At its small final learn­ing rate a decayed model barely moves. At a higher rate it learns the new domain and for­gets the old. The fix that works at scale is cheap: raise the rate again, decay it again, and keep some old data in every batch [358, 526]. Gains are largest for domains far from the orig­i­nal data, as with RoBERTa [527]. Modern runs read 50B to 6T tokens, where the sched­ule and for­get­ting decide the out­come.

The recipe: re-warm, re-decay, replay

Ibrahim et al. tested the recipe on Pile-pre­trained models moved to SlimPajama (a weak shift, at 405M and 10B) or German web text (a strong shift, at 405M only) [358]. Re-warm lin­early over about 1% of the new steps to the orig­i­nal peak learn­ing rate. Re-decay with cosine to a tenth of that peak inside the new budget. Replay a frac­tion $x$ of every batch from the old data. The opti­mizer state can be reset, since load­ing it made a neg­li­gi­ble dif­fer­ence [528]. Replay is com­pute-equiv­a­lent, because old tokens replace new ones:

With $D_{\text{CPT}}=300$B tokens and $x=5\%$, the run sees 285B new and 15B replayed tokens. For the 10B model (9.6B param­e­ters) that is $6\times 9.6\times {10}^9\times 3\times {10}^{11}=1.73\times {10}^{22}$ FLOPs, half the $3.46\times {10}^{22}$ of retrain­ing from scratch on the 600B-token union (com­puted).

Figure 39 plots what replay bought the 405M model. Follow the old-data (Pile) line: it drops steeply between 0% and 1% replay and then flat­tens, while the new-data line hardly rises. With­out replay the German shift raised the Pile loss by 1.39 nats (2.17 to 3.56), against 0.27 for the weak shift. Replay of 25% removed 1.23 of it and matched the aver­age loss of a model retrained on both datasets [358], which reads 2.5 times the CPT tokens (500B against 200B, com­puted). At 10B, 5% replay reached an aver­age loss of 1.89, against 1.87 for the union and 1.99 with none [358].

When to re-warm

Part of what looks like for­get­ting is the re-warm itself. With no shift at all, re-warm­ing to $3\times {10}^{-4}$ raised the Pile loss by 0.1 at its peak, against 0.35 when switch­ing to SlimPajama [358]. The anneal­ing law of Sec­tion 7.1 reads a re-warm as a neg­a­tive anneal­ing area [359], and a higher peak trades more adap­ta­tion for more for­get­ting [526]. That is one reason to keep a pre-decay check­point (Sec­tion 5.2). DeepSeekMath started from one [207].

Parmar et al. argue the other side for a gen­eral refresh. Con­tin­u­ing a 15B model pre­trained on 8T tokens on mostly old sources plus QA data, their best sched­ule started at the pre­train­ing min­i­mum learn­ing rate, and accu­racy rose from 48.9 to 55.0 after 100B tokens but only to 56.8 after 1T [528]. The two results agree once you sort by the data. Re-warm for large, dif­fer­ent data. Stay near the min­i­mum rate for mostly old data reweighted, as in the decay-phase mixes of Sec­tion 7.2.

Other defenses against for­get­ting

Elas­tic weight con­sol­i­da­tion (EWC) ties each param­e­ter to its old value with a spring whose stiff­ness is the diag­o­nal Fisher infor­ma­tion $F_i$ of the old task [529]:

At LLM scale it rarely pays: $F$ and ${\theta }^{\star }$ in FP32 for Llama 3 8B take $2\times 8.03\times {10}^9\times 4$ bytes, about 64 GB (com­puted), and $F$ needs old data anyway. The table lists the cheaper levers.

Domain models built this way

Table 7.6 com­pares six pub­lished domain models. Read the start­ing point, the token count and the replay share together, and check which results count regres­sions as well as gains.

The recipes share a pat­tern: the strong­est related base, often from before its decay, tens to hun­dreds of bil­lions of clas­si­fier-mined domain tokens (Sec­tion 3.2), 5⁠–⁠30% gen­eral or replay data, a re-decay inside the run, and eval­u­a­tion of regres­sions as well as gains.

A new lan­guage that the tok­enizer serves badly costs more tokens per doc­u­ment (Sec­tion 3.7). Vocab­u­lary exten­sion appends rows to the input embed­ding and, if untied, the output head, adding $\Delta P=(V^{\prime }-V)d$ param­e­ters tied or $2(V^{\prime }-V)d$ untied. Chi­nese LLaMA grew its vocab­u­lary from 32,000 to 49,953, roughly halv­ing the tokens per Chi­nese text [537], for $2\times \mathrm{17,953}\times \mathrm{4,096}=147$M param­e­ters, about 2% of the 7B (com­puted).

How to do it your­self. Start from a pre-decay check­point if you have one, run domain and reten­tion suites before and after, and expect an early rise in old-data loss. Alert only if it does not recover (Sec­tion 12.5).

7.7 Smaller models from bigger ones: distillation and pruning

How do you get a strong small model from a big one with a frac­tion of the tokens, and when is the teacher worth its cost? A hard next-token label is one sample from a broad dis­tri­bu­tion. A teacher's full dis­tri­bu­tion also says which wrong tokens were nearly right: "When the soft tar­gets have high entropy, they pro­vide much more infor­ma­tion per train­ing case than hard tar­gets and much less vari­ance in the gra­di­ent between train­ing cases" [538]. Dis­til­la­tion trains a stu­dent to match that dis­tri­bu­tion. Prun­ing also starts the stu­dent as a trimmed copy of the teacher.

The loss and its gra­di­ent

Take stu­dent logits $z$ and teacher logits $v$, soft­ened by a tem­per­a­ture $\tau$ (sub­scripts S and T mean stu­dent and teacher): $q=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(z/\tau )$, $p=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(v/\tau )$. The soft-target loss $-{\sum }_j{p}_j\log q_j$ is $\mathrm{K}\mathrm{L}(p\Vert q)$ plus a con­stant. Since $\partial \log q_j/\partial z_i=({\delta }_{ij}-q_i)/\tau$,

At high tem­per­a­ture $q_i-p_i$ itself shrinks as $1/\tau$, so the gra­di­ent falls as $1/{\tau }^2$ and the soft loss is mul­ti­plied by ${\tau }^2$ [538]. A finite-dif­fer­ence check agrees to ${10}^{-9}$ (com­puted). LLM pre­train­ing uses $\tau =1$, which Bus­bridge et al. found opti­mal [471]. This for­ward KL is mode-cov­er­ing. For gen­er­a­tion, MiniLLM min­i­mizes reverse KL [539], GKD trains on the stu­dent's own sam­ples [540], and sequence-level dis­til­la­tion trains on teacher out­puts [541], as rea­son­ing-trace dis­til­la­tion does (Sec­tion 9.4).

Dis­til­la­tion pre­train­ing in prac­tice

Sev­eral open fam­i­lies pre­train small models this way. Gemma 2's abla­tion iso­lates the effect: a 2B model trained on 500B tokens aver­aged 67.7 on three bench­marks with a 7B teacher, against 60.3 from scratch (reported) [268].

What dis­til­la­tion costs, and when it pays

The stu­dent pays its usual $6N_S$ FLOPs per token. Online logits add a teacher for­ward pass, $2N_T$, and a teacher trained only for this stu­dent adds its whole pre­train­ing, the term in paren­the­ses [471]:

For a 9.24B stu­dent and a 27B teacher the share is $54/(55.4+54)=49\%$, so dis­til­la­tion dou­bles the bill (com­puted). The 27B is illus­tra­tive, since Gemma 2 does not name the 9B's teacher [268]. To cut the share, use a smaller teacher or store sparse logits once and reuse them. The top 256 entries at a 4-byte index and a 2-byte value take 1,536 bytes per token, 6.1 PB for 4T tokens (com­puted, illus­tra­tive format), and Mini­tron lost sig­nif­i­cant accu­racy keep­ing 100 or fewer [495].

The dis­til­la­tion scal­ing law says when it pays. With many stu­dents or an exist­ing teacher, dis­til­la­tion beats super­vised learn­ing up to a com­pute level that grows pre­dictably with stu­dent size. With one stu­dent and a teacher trained for it, super­vised learn­ing is gen­er­ally better, and a teacher that is too capa­ble even­tu­ally hurts [471]. A 2026 study finds that even small, under­trained teach­ers help larger stu­dents [543].

Prun­ing plus dis­til­la­tion: Mini­tron

Mini­tron scores every atten­tion head, MLP neuron, embed­ding chan­nel and layer from for­ward acti­va­tions on 1,024 cal­i­bra­tion sam­ples, slices the weights to the target width or depth, and retrains on the unpruned model's logits alone [495]. Prun­ing in stages (15B to 8B to 4B) beat one shot by 12% on MMLU. The 8B and 4B each used 94B tokens, up to 40 times fewer than from scratch, for a 1.8x saving on the family. That counts stu­dent train­ing only, at per-step FLOPs of 4.4, 2.5 and $1.2\times {10}^{17}$ [495]:

Count­ing the teacher's for­ward passes, a third of a 15B train­ing step per dis­til­la­tion token, gives about 1.77x (com­puted). Qual­ity held: Mini­tron 8B scored 63.8 on MMLU against 54.7 for Nemotron-3 8B trained from scratch on 3.8T tokens [495].

Llama-3.1-Mini­tron works with­out the orig­i­nal pre­train­ing data [544]. It fine-tunes the teacher on about 100B tokens of the dis­til­la­tion data first (teacher cor­rec­tion, which cut val­i­da­tion loss by over 6%), then prunes Llama 3.1 8B to 4.5B and dis­tills on 94B tokens, 160 times fewer than the parent's 15T (com­puted, the paper says 150). Figure 40 opens on this pipe­line: com­pare the two dots at the top, then watch the stacked bar as you raise the teacher size or switch on teacher train­ing.

Sheared LLaMA learns prun­ing masks on 0.4B tokens and con­tin­ues on 50B [545], and layer drop­ping with light heal­ing removes up to roughly half of Llama-2⁠–⁠70B's layers [274].

A 3B model trains with batches of 512 sequences of 4,096 tokens. How many tokens give a decay as long in steps as OLMo 2 7B's 50B-token anneal?

One step is $512\times \mathrm{4,096}=\mathrm{2,097,152}$ tokens, and OLMo 2's anneal is 11,921 steps, so about 25.0B tokens (com­puted). The sched­ule is defined in steps, which is why OLMo 2 gave its 13B, with twice the batch, a 100B anneal [7].

A soup of three anneals keeps 50% of one run's noise excess loss. What pair­wise cor­re­la­tion does the toy model imply, and what would a fourth anneal give?

$(1+2\rho )/3=0.5$ gives $\rho =0.25$. With $k=4$, $(1+3\times 0.25)/4=44\%$, and no number of ingre­di­ents goes below $\rho =25\%$ (com­puted). Gains need decor­re­lated ingre­di­ents, such as anneals with dif­fer­ent data orders [7, 400].

Llama 3 8B has base 500,000 and $d_h=128$ and trains at 8,192 tokens. Under YaRN with ${\alpha }_{\text{Y}}=1$ and ${\beta }_{\text{Y}}=32$, how many rotary pairs are untouched, blended and fully inter­po­lated?

Untouched pairs make over 32 turns, so ${\lambda }_j$ is under 256 tokens: pairs 0⁠–⁠18 (19). Fully inter­po­lated pairs make under one turn, so ${\lambda }_j$ is over 8,192: pairs 35⁠–⁠63 (29). Pairs 19⁠–⁠34 (16) blend (com­puted from ${\lambda }_j=2\pi ·500,{000}^{2j/128}$) [507].

With 25% com­pute-equiv­a­lent replay on the 200B-token German shift, how many German tokens does the 405M model see, and what share of the 1.39-nat for­get­ting does replay remove?

150B German and 50B Pile tokens. Replay removed 1.23 of the 1.39 nats, $1.23/1.39=88\%$ (com­puted), and matched the aver­age loss of retrain­ing on both datasets [358].

Four 1.2B stu­dents share one set of stored logits from an 8B teacher. What share of each stu­dent's FLOPs is the teacher's?

The teacher's for­ward pass, $2\times 8.03\times {10}^9$ FLOPs per token, is paid once and split four ways: $4.0\times {10}^9$ per stu­dent token against $7.2\times {10}^9$ of stu­dent train­ing, so 36%, against 69% for one stu­dent (com­puted, stor­age ignored). Reuse is how dis­til­la­tion pays [471].

8. Post-training I: instruction tuning, preferences and RLHF

Post-train­ing changes which behav­iors a pre­trained model reli­ably pro­duces. Demon­stra­tions teach the response; pref­er­ences rank alter­na­tives; reward opti­miza­tion changes what the policy sam­ples. The data format and the exact loss decide which of those lessons reaches the weights.

A base model, a pre­trained model before any fur­ther train­ing, pre­dicts the next token of web text. To it a ques­tion is only a prefix, and noth­ing in pre­train­ing says "answer help­fully, then stop". Llama 3 defines post-train­ing as "any model train­ing that hap­pens out­side of pre-train­ing" [6], and Lam­bert frames it as elic­it­ing behav­iors the base model already has [547]. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

Label­ers pre­ferred out­puts of the 1.3B InstructGPT model to out­puts of the 175B GPT-3, a model with over 100 times more param­e­ters [8]. The 175B InstructGPT was pre­ferred to 175B GPT-3 85 ± 3% of the time, and to a few-shot prompted 175B GPT-3 71 ± 4% of the time [8] (reported). Train­ing the 175B SFT model took 4.9 PF/s-days (petaFLOP/s-days) and the 175B PPO-ptx model 60, against 3,640 for GPT-3 [8]: 0.13% and 1.6% of the pre­train­ing com­pute. DeepSeek-V3 spent 0.18% of its GPU-hours on post-train­ing, research and abla­tions excluded [2] (all com­puted in Sec­tion 0.2). Gen­er­at­ing data is cheap too: a mil­lion Llama 3 405B rejec­tion-sam­pling prompts at $n=30$ cost about 0.075% of the pre­train­ing run (com­puted in Sec­tion 8.8). Post-train­ing bud­gets are set by people and iter­a­tion speed, not by FLOPs.

InstructGPT's three stages, fed from one prompt pool, are the skele­ton: super­vised fine-tuning on about 13k prompts, a reward model on 33k and PPO on 31k [8]. Later recipes swap parts: DPO replaces PPO, ver­i­fi­able rewards replace the reward model for math and code (Chap­ter 9), and the loop repeats, six rounds in Llama 3 [6]. Step through Figure 41 and watch the memory bar jump when a stage holds sev­eral models.

The chap­ter fol­lows the chain: imi­ta­tion in Sec­tions 8.1 to 8.3, rewards and their fail­ures in 8.4 to 8.6, then DPO, recipes and safety in 8.7 to 8.9.

8.1 What post-training teaches: format, skills and the superficial alignment hypothesis

If pre­train­ing already put the knowl­edge in, what is left for post-train­ing to teach, and how much data does it take? LIMA fine-tuned LLaMA 65B with the stan­dard super­vised loss on 1,000 curated prompts and responses, about 750,000 tokens, with no rein­force­ment learn­ing or pref­er­ence model [548]. Its authors stated the super­fi­cial align­ment hypoth­e­sis: "A model’s knowl­edge and capa­bil­i­ties are learnt almost entirely during pre­train­ing, while align­ment teaches it which sub­dis­tri­bu­tion of for­mats should be used when inter­act­ing with users" [548]. Raters found LIMA's answers equiv­a­lent or pre­ferred to GPT-4's in 43% of cases [548] (reported). The com­pute for $E$ epochs over $D_{\text{SFT}}$ tokens, masked prompt tokens included, is

LIMA trained 15 epochs [548]: $6\times 6.5\times {10}^{10}\times 7.5\times {10}^5\times 15=4.4\times {10}^{18}$ FLOPs, against $6\times 6.5\times {10}^{10}\times 1.4\times {10}^{12}=5.5\times {10}^{23}$ for pre­train­ing on 1.4T tokens [46], a ratio of $8\times {10}^{-6}$ (com­puted).

Token-level evi­dence agrees. On Llama 2 chat's answers, 77.7% of tokens were also the base model's top choice and 92.2% were in its top three, and the few shifted tokens were mostly stylis­tic [549].

The strong form still breaks. Skills keep scal­ing with data: Tülu 3's models "con­tinue to improve on aver­age as more SFT data is included" [550]. And they depend on the base: the same mix gives GSM8K and MATH scores of 76.2 and 31.5 on Llama 3.1 8B, 91.1 and 53.7 on Llama 3.1 70B, and 86.3 and 56.4 on Qwen 2.5 Math 7B [550] (reported).

An older lin­eage con­verted aca­demic datasets into instruc­tions: FLAN on over 60 datasets [551], T0 [552], Super-NaturalInstructions with 1,616 tasks [553] and the Flan col­lec­tion with 1,836 [554]. Whether this helps small models is con­tested. FLAN found it hurt at 8B and below [551], while the Flan col­lec­tion reports held-out gains rising log-lin­early with task count [555]. Our untested read­ing is that the mix­ture made the dif­fer­ence.

Chat data won, and InstructGPT shows why. On real API prompts it won 73.4 ± 2% of com­par­isons against the base­line, against 26.8 ± 2% and 29.8 ± 2% for GPT-3 tuned on T0 and FLAN data, because clas­si­fi­ca­tion and QA were only about 18% of API use [8]. Imi­ta­tion has a ceil­ing too: models tuned on ChatGPT out­puts "close little to none of the gap from the base LM to ChatGPT on tasks that are not heav­ily sup­ported in the imi­ta­tion data" [556]. Eval­u­ate SFT on capa­bil­ity, not only on style (Chap­ter 10).

8.2 Supervised fine-tuning in practice: data, templates and the loss

What exactly do you train on in SFT, and which imple­men­ta­tion details silently change the result? Super­vised fine-tuning (SFT), or instruc­tion tuning, is next-token train­ing on curated prompt and response pairs with the loss on the responses only. Its data, token format and loss book­keep­ing each hide a fail­ure mode.

Where the data comes from

SFT data has had three eras. First, human demon­stra­tions: InstructGPT trained on 12,725 SFT prompts, 11,295 writ­ten by label­ers, so 88.8% (com­puted) [8]. Llama 2 stopped at 27,540 anno­ta­tions because its own SFT model's sam­ples "were often com­pet­i­tive with SFT data hand­writ­ten by human anno­ta­tors" [229]. Second, dis­til­la­tion from stronger models: Self-Instruct grew 175 seed tasks into about 52k instruc­tions [557], Vicuna trained on 70K shared ChatGPT chats [558], and over 98% of Nemotron-4 340B's align­ment data was syn­thetic [291]. Third, sam­pling from the model being trained and fil­ter­ing, as Llama 3 does every round [6] (rea­son­ing traces are in Sec­tion 9.4).

Choice beats volume. In QLoRA's study, more data moved MMLU by at most 0.5 points while the choice of dataset moved it by up to 8 [76].

A worked mix: Tülu 3

Tülu 3's SFT mix holds 939,344 prompts [550], decon­tam­i­nated (cleaned of bench­mark over­lap) by 8-gram match­ing on prompts [550] (Sec­tion 10.5). Remov­ing one data source at a time from the 8B mix (single runs) cut IFEval from 72.8 to 53.6 with­out the per­sona data, GSM8K from 76.2 to 64.1 and MATH from 31.5 to 23.5 with­out the math data, the safety aver­age from 93.1 to 74.7 with­out the safety data, and AlpacaEval 2 from 12.4 to 7.5 with­out WildChat [550]. For scale, five 8B train­ing seeds span 59.8 to 60.1 on the aver­age score and three 70B seeds span 70.0 to 72.6 [550]. The paper reports seed spread only for the aver­age, so single-bench­mark changes of a point or two are not clearly signal. In Figure 42, look at the out­lined cells: each slice pro­tects its own skill while other cells move by less than 3 points.

Chat tem­plates and loss mask­ing

A chat tem­plate is the exact token format of a con­ver­sa­tion. Llama 3 uses spe­cial header and ter­mi­na­tion tokens [6], with <|eot_id|> ending each mes­sage [559]. Loss mask­ing zeroes the loss on tokens the model should not pro­duce, so here only the last line car­ries loss.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Be concise.<|eot_id|><|start_header_id|>user<|end_header_id|>

Capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Paris.<|eot_id|>   # loss here only

Render train­ing data with add_generation_prompt=False [560] and serve with the tem­plate you trained on: tem­plate vari­ants alone moved Tülu 3's aver­age between 51.6 and 53.0 [550]. Mask­ing prompts is the norm [6], with one excep­tion: for long instruc­tions with short out­puts, loss on the instruc­tions helped [561]. A Llama 3 SFT exam­ple aver­ages 846.1 tokens with 310.4 in the final response [6], so only $310.4/846.1=37\%$ of the tokens you pay for carry loss (com­puted).

Pack­ing and the loss

Pack­ing con­cate­nates exam­ples into fixed-length rows to avoid pad­ding. They must stay inde­pen­dent, which DeepSeek-V3 does with "a sample mask­ing strat­egy to ensure that these exam­ples remain iso­lated and mutu­ally invis­i­ble" [2]: a block-diag­o­nal atten­tion mask and posi­tion ids that restart per exam­ple. With the mask $m_t=1$ on assis­tant tokens, the loss is

Both sums run over the whole global batch, every micro-batch and data-par­al­lel rank. Loss aggre­ga­tion, how token losses are com­bined, sets each exam­ple's weight. Tülu 3 traced a gap between two train­ers to "Aver­ag­ing the loss across pad­ding tokens with­out taking into account gra­di­ent accu­mu­la­tion or dis­trib­uted train­ing setups" [550]. For two exam­ples with summed losses ${\ell }_1,{\ell }_2$ and token counts $n_1,n_2$:

Token weights are $1/(n_1+n_2)$ in the first and $1/(2n_i)$ in the second, so "in the second case we weight each exam­ple equally, while in the first we weight each token equally" [550].

The fix is a few lines, and its unit test is that one micro-batch of eight sequences and eight micro-batches of one give the same gra­di­ent.

import torch.nn.functional as F

def train_step(model, opt, micro_batches):
    # labels: input_ids shifted left by one, -100 on non-assistant tokens
    Z = sum(int((mb["labels"] != -100).sum()) for mb in micro_batches)
    # data parallel: all-reduce (sum) Z over ranks, and divide by
    # Z / p_dp if the framework averages gradients over ranks
    for mb in micro_batches:
        logits = model(mb["input_ids"]).logits     # [b, s, V]
        nll = F.cross_entropy(logits.flatten(0, 1), mb["labels"].flatten(),
                              ignore_index=-100, reduction="sum")
        (nll / Z).backward()     # weight 1/Z per loss token
    opt.step()
    opt.zero_grad()

Hyper­pa­ram­e­ters and check­point selec­tion

SFT learn­ing rates sit about 8 to 60 times below pre­train­ing peaks: Llama 3 405B pre­trains at $8\times {10}^{-5}$ and its largest models fine-tune at ${10}^{-5}$ [6], while Tülu 3 fine-tunes Llama 3.1 8B, pre­trained at a peak of $3\times {10}^{-4}$, at $5\times {10}^{-6}$ [6, 550] (ratios com­puted). Table 8.1 col­lects pub­lished set­tings, which trans­fer between train­ers only if loss reduc­tion, batch and sequence length match. Tülu 3's 8B run took 32 GPUs for 6 hours [550], 192 GPU-hours (com­puted).

Do not pick check­points by val­i­da­tion loss. InstructGPT's SFT models "over­fit on val­i­da­tion loss after 1 epoch", yet more epochs improved human pref­er­ence [8], and LIMA found that "per­plex­ity does not cor­re­late with gen­er­a­tion qual­ity" [548]. Select on sam­pled answers instead.

Two more traps. Judge-only gains may be length, as with NEFTune's jump from 29.79% to 64.69% on AlpacaEval [563] (Sec­tion 10.6). And for­get­ting tracks the KL shift from the base [564], so mix gen­eral data into domain SFT.

How to do it your­self. Start from an open mix such as Tülu 3's, decon­tam­i­nate against your bench­marks, render with your serv­ing tem­plate, mask non-assis­tant tokens, pack with per-exam­ple masks, nor­mal­ize over the global batch, train 2 epochs near $5\times {10}^{-6}$ at batch 128 for an 8B model, run two or three seeds and select by gen­er­a­tion qual­ity.

8.3 Parameter-efficient fine-tuning: LoRA, QLoRA and merging

Can you adapt a model by train­ing a tiny frac­tion of its weights, and what do you give up? Full fine-tuning with mixed-pre­ci­sion Adam holds about 16 bytes per param­e­ter (Sec­tion 1.4), 119.7 GiB for Llama 3 8B and 1,051 GiB for 70B (com­puted). Low-rank adap­ta­tion (LoRA) trains a small update instead, and on GPT-3 175B it cut train­able param­e­ters 10,000 times and train­ing memory from 1.2 TB to 350 GB [565].

The mech­a­nism and its arith­metic

Freeze the weight $W_0$ and learn a rank-$r$ update. In this sec­tion $r$ is the LoRA rank, not a reward, and $\alpha$ is a scale con­stant:

$A$ starts Gauss­ian and $B$ at zero, so train­ing starts exactly at the base model, and the authors "simply set α to the first r we try and do not tune it" [565]. After train­ing, fold $BA$ into $W_0$ and serve with no extra latency. Each adapted matrix adds $r(d_{\text{in}}+d_{\text{out}})$ param­e­ters [565]:

Llama 3 8B has $d=\mathrm{4,096}$, $d_{\text{ff}}=\mathrm{14,336}$ and 8 KV heads of 128, so K and V project to 1,024 [6, 17]. Atten­tion adds $r(\mathrm{8,192}+\mathrm{5,120}+\mathrm{5,120}+\mathrm{8,192})=\mathrm{26,624}r$ per layer and the MLP $3r(\mathrm{4,096}+\mathrm{14,336})=\mathrm{55,296}r$, so $\mathrm{81,920}r$ in all. At rank 16 over 32 layers that is $\mathrm{81,920}\times 16\times 32=41.9$M param­e­ters, 0.52% of the 8.03B (com­puted). This repro­duces Think­ing Machines' four Llama 3.1 8B counts of 0.49B, 0.25B, 0.70B and 0.24B [566] only if the output head is also adapted, which the post does not state (com­puted).

FLOPs fall by about a third, not 99%, because a frozen matrix skips only its weight gra­di­ent: $2d^2+6dr$ mul­ti­ply-adds for a $d\times d$ matrix against $3d^2$, slightly more than two thirds [566]. With $N_{\text{mm}}$ weights in matrix mul­ti­plies (linear layers and output head), LoRA costs $4N_{\text{mm}}+6P_{\text{LoRA}}$ FLOPs per token against $6N_{\text{mm}}$:

That is 0.675 for a 4,096-wide square matrix at rank 16 and $2/3+41.9\text{M}/\mathrm{7,505}\text{M}=0.672$ for the whole 8B model, ignor­ing atten­tion scores (com­puted). Mea­sured on GPT-3 175B: 43.1 against 32.5 tokens/s per V100 [565], 1.33 times faster (com­puted).

When it matches full fine-tuning

Adapt every linear layer. QLoRA found "LoRA on all linear trans­former block layers are required to match full fine­tun­ing per­for­mance" [76], and Think­ing Machines that "Atten­tion-only LoRA sig­nif­i­cantly under­per­forms MLP-only LoRA". Con­fig­ured that way, "LoRA per­forms the same as full fine-tuning" for small-to-medium instruc­tion and rea­son­ing sets, though it tol­er­ates large batches less well, and for policy-gra­di­ent RL it matches "even with ranks as low as 1" [566]. The limit: full fine-tuning wins at con­tin­ued pre­train­ing on code and math and learns updates of 10 to 100 times higher rank, while LoRA for­gets less [530].

The opti­mal LoRA learn­ing rate is about 10 times the full fine-tuning rate, with a fitted mul­ti­plier of 9.8, at $\alpha =32$ with the $\alpha /r$ scale [566]. Under $\alpha /r$ the best rate barely moves with rank [566], and rsLoRA argues for $\alpha /r^{1/2}$ instead [567].

QLoRA

QLoRA trains LoRA adapters on a frozen 4-bit base. Reg­u­lar 16-bit fine-tuning of a 65B model needs more than 780 GB of GPU memory, and QLoRA brings it under 48 GB [76]. NF4 stores each weight as one of 16 levels at normal quan­tiles, with a scale per block of 64. Double quan­ti­za­tion cuts those scales from 0.5 to 0.127 bits per param­e­ter, and paged opti­miz­ers absorb memory spikes [76]:

The Llama 3 70B base then takes $70.55\times {10}^9\times 4.127/8$ bytes = 33.9 GiB if every param­e­ter is quan­tized, or about 36.8 GiB if the embed­ding and output head stay in 16-bit (com­puted). QLoRA's defaults are rank 64, $\alpha =16$ and all linear layers [76]. DoRA splits each pre­trained weight into a mag­ni­tude and a direc­tion, trains both and applies LoRA to the direc­tion [568]. None of this chap­ter's recipes use it. In Figure 43, try the "QLoRA on 70B" preset and watch the bar drop under the 80 GB line.

Merg­ing

A task vector is fine-tuned minus pre­trained weights. Adding task vec­tors com­bines skills, and negat­ing a tox­i­c­ity vector cut toxic gen­er­a­tions from 4.8% to 0.8% [569]. TIES-Merg­ing aver­ages only each vector's top 20% of entries whose signs agree [570]. Llama 3 aver­ages vari­ants at each reward-model, SFT and DPO stage [6], while Tülu 3's best soup only matched its best seed [550]. Weight aver­ag­ing and soups are explained in Sec­tion 5.7.

The usual mis­takes:

• Adapters on atten­tion only.
• The full fine-tuning learn­ing rate. Start near 10 times it.
• Very large batches.
• Expect­ing an adapter to absorb con­tin­ued-pre­train­ing vol­umes of data.
• Merg­ing an adapter into a quan­tized base, which changes the numer­ics you eval­u­ated.
• Merg­ing check­points with dif­fer­ent tok­eniz­ers or chat tem­plates.

8.4 Preferences and reward models

How do you turn "A is better than B" into a number a model can opti­mize, and how good can that number be? You train a second net­work, the reward model, so that the gap between its scores for two responses pre­dicts which one a person would pick. Com­par­ing is easier than demon­strat­ing: a labeler who could not write the ideal answer can still tell which of two drafts is shorter and still cor­rect.

Chris­tiano et al. learned con­trol rewards from com­par­isons on "less than 1% of our agent’s inter­ac­tions" and taught a sim­u­lated hopper a back­flip with 900 queries [571]. Sti­en­non et al. found that dou­bling the com­par­i­son data added about 1.1% reward-model accu­racy and dou­bling the model about 1.8% (reported) [572], and Anthropic found that "ranked pref­er­ence mod­el­ing per­forms much better than imi­ta­tion learn­ing" [573].

The Bradley-Terry model

Give every response a hidden strength. The Bradley-Terry model says the chosen response $y_w$ beats the rejected response $y_l$ with prob­a­bil­ity equal to the logis­tic func­tion $\sigma$ of the strength gap [574, 571]. The reward model $r_{\varphi }$ sup­plies the strengths and is fit by max­i­mum like­li­hood:

where $\Delta r=r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$ is the reward margin. In prac­tice $r_{\varphi }$ is the SFT trans­former with a scalar head, read at the last token [8]. The table shows two prop­er­ties. The gra­di­ent shrinks as the margin grows, so con­fi­dently ordered pairs stop teach­ing. And only dif­fer­ences enter the loss, so the reward has no fixed zero and is recen­tered or whitened before RL.

Col­lect­ing com­par­isons

InstructGPT label­ers ranked 4 to 9 responses per prompt, so a rank­ing of $K$ responses yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$, or 6 to 36, com­par­isons (com­puted). All com­par­isons from one prompt share a batch ele­ment, up to 2,304 per batch, train­ing runs one epoch because "mul­ti­ple epochs quickly over­fit the model", and a 6B reward model served even the 175B policy [8]. At 2,000 com­par­isons Gao et al.'s reward models reached a val­i­da­tion loss of 0.686 against a coin flip's $\ln 2=0.693$ (reported) [575].

Llama 2 trained sep­a­rate help­ful­ness and safety reward models on 1,418,091 com­par­isons, with a margin term that grows with the labeler's stated pref­er­ence strength. It raised accu­racy on "sig­nif­i­cantly better" pairs from 79.1% to 80.7% but the aver­age only from 62.5% to 62.9% (reported) [229]. Llama 3 dropped the margin, citing "dimin­ish­ing improve­ments after data scal­ing", and kept only pairs labeled "sig­nif­i­cantly better" or "better" [6]. Reward-model accu­racy decays as the policy moves, so each round labels fresh sam­ples [229].

How much do label­ers agree?

Most InstructGPT com­par­isons had a single labeler, for cost rea­sons. Train­ing label­ers agreed with each other 72.6 ± 1.5% of the time, and the reward models pre­dicted train­ing-group label­ers at 72.4 ± 0.4% and held-out label­ers at 69.6 ± 0.9% (reported) [8]. Is that near the ceil­ing? Sup­pose each label is the true pref­er­ence flipped inde­pen­dently with prob­a­bil­ity $q$. Two label­ers agree with prob­a­bil­ity $a=(1-q{)}^2+q^2$, so

Is an accu­rate reward model a good teacher?

RewardBench scores reward models on 2,985 prompt, chosen and rejected trios across chat, rea­son­ing and safety, where guess­ing scores 50% [576, 577]. The lead­ers are curated: HelpSteer2 reached 92.0% with ten thou­sand pairs [577]. But accu­racy is not teach­ing qual­ity. Razin et al. show that "even a per­fectly accu­rate reward model can lead to extremely slow opti­miza­tion" if its rewards barely vary across the policy's sam­ples [578], and mod­er­ately accu­rate reward models have trained better poli­cies than highly accu­rate ones [579].

When the labeler is a model

Con­sti­tu­tional AI has a model apply writ­ten prin­ci­ples to label harm­less­ness, and mixed 182,831 such AI com­par­isons with 135,296 human help­ful­ness ones [580]. RLAIF matched RLHF on sum­ma­riza­tion (71% against 73% pre­ferred over SFT) and dia­logue (63% against 64%) at over 10x lower label­ing cost (reported) [581]. Gen­er­a­tive reward models write a judg­ment before scor­ing: GenRM lifted best-of-N on GSM8K from 73% to 93.4% [582], though zero-shot judges trail Bradley-Terry models in dis­tri­bu­tion by 9⁠–⁠36% [583]. Newer work writes rewards as rubrics or check­lists [584, 585]. Judge biases are in Sec­tion 10.6.

8.5 Optimizing the reward: best-of-n, rejection sampling and PPO

Given a reward model, how do you move the policy toward higher reward with­out break­ing it, and what does each method cost? Every method here opti­mizes expected reward minus a penalty for drift­ing from the ref­er­ence policy ${\pi }_{\text{ref}}$, usu­ally the SFT model [586],

with $\beta$ the KL coef­fi­cient. The penalty keeps text fluent and the policy where the reward model was trained. InstructGPT used $\beta =0.02$ for 256k episodes, plus a pre­train­ing log-like­li­hood term (PPO-ptx) against bench­mark regres­sions [8].

Best-of-n: no train­ing at all

Sample $n$ responses and return the one the reward model likes best. With con­tin­u­ous rewards and dis­tinct sam­ples, the winner's den­sity is $n{F}^{n-1}$ times the ref­er­ence den­sity, where $F$ is the reward's cumu­la­tive dis­tri­bu­tion, and $F$ of the winner has $𝔼[\ln F]=-1/n$. So [572, 575]

which is 0.19 nats at $n=2$, 1.84 at 16, 5.91 at 1,000 and 10.0 at 60,000 (com­puted). It grows like $\ln n$, so best-of-$n$ is a cau­tious opti­mizer. The for­mula is an upper bound [587]: with two equally likely responses no selec­tion can move more than $\ln 2=0.69$ nats, while the for­mula gives 1.40 at $n=10$ (com­puted).

Rejec­tion-sam­pling fine-tuning

Rejec­tion-sam­pling fine-tuning moves best-of-$n$'s serv­ing cost into train­ing: sample, keep the best, fine-tune on the win­ners, repeat [588, 589]. Llama 2 used only this until its fourth RLHF ver­sion, then added PPO on top [229]. Llama 3 sam­ples $n$ (the paper's $K$) between 10 and 30 responses per prompt [6], so the kept responses sit at most 1.40 to 2.43 nats from the sam­pling policy (com­puted). That bounds the selec­tion step, not the model fine-tuned on it.

PPO in its RLHF form

Prox­i­mal policy opti­miza­tion (PPO) treats gen­er­a­tion as an episode. The state is the prompt plus the tokens so far, $(x,y_{<t})$, the action is the next token, and the reward model's score arrives at the end-of-sequence (EOS) token. The KL penalty is spread over tokens as log-ratios:

Summed over tokens this is a one-sample esti­mate of the objec­tive. That log-ratio esti­ma­tor is unbi­ased but "neg­a­tive for half of the sam­ples", while $k_2=\frac{1}{2}(\log ({\pi }_{\theta }/{\pi }_{\text{ref}}){)}^2$ has "much lower vari­ance" but is biased [590]. His $k_3={\pi }_{\text{ref}}/{\pi }_{\theta }-1-\log ({\pi }_{\text{ref}}/{\pi }_{\theta })$ is unbi­ased and had lower spread still in his test [590]. PPO max­i­mizes a clipped sur­ro­gate, "a pes­simistic bound on the unclipped objec­tive" [591], with advan­tages ${\widehat{A}}_t$ from a value model $V_{\psi }$ through gen­er­al­ized advan­tage esti­ma­tion (GAE) [592]:

RLHF runs use $ϵ=0.2$, dis­count $\gamma =1$ and $\lambda =0.95$, and ini­tial­ize the value model from the reward model [593, 575]. Sec­tion 9.2 derives the policy gra­di­ent behind this.

Four models and their cost

PPO-RLHF "typ­i­cally needs to load 3⁠–⁠4 models into the GPU memory" [593]: the trained policy and value model, and the frozen ref­er­ence and reward model. With all four of size $N$ and $E$ PPO epochs, each gen­er­ated token costs four for­wards at $2N$ and, per epoch, two train­ing passes at $6N$:

plus $2N$ if a sep­a­rate infer­ence engine gen­er­ates and the policy re-scores the roll­outs.

The details that decide whether it works

"The reward scores and train­ing losses do not indi­cate whether the PPO is opti­miz­ing cor­rectly", so Zheng et al. mon­i­tor per­plex­ity, response length and KL to the SFT model. Their col­lapsed runs gave longer responses with lower per­plex­ity [596].

This fragility is why Llama 3 chose rejec­tion sam­pling and DPO over RL meth­ods "that tend to be less stable and harder to scale" [6]. Sec­tion 8.8 weighs the evi­dence.

8.6 When the reward is wrong: over-optimization, length bias and sycophancy

How far can you opti­mize a learned reward before true qual­ity falls, and what does the policy learn when the reward is subtly wrong? A reward model is a proxy. Push against it and the proxy keeps rising while true qual­ity rises, peaks and falls (Good­hart's law).

Gao, Schul­man and Hilton let a fixed "gold" reward model, the 6B InstructGPT one, stand in for humans. It labeled 100,000 syn­thetic com­par­isons, proxy reward models of 3M to 3B param­e­ters learned from them, and a 1.2B policy was opti­mized against each proxy by best-of-$n$ and by PPO with no KL penalty [575]. With $\kappa =D_{\text{KL}}(\pi \Vert {\pi }_{\text{init}}{)}^{1/2}$, the dis­tance in which Bai et al. found reward to grow roughly lin­early [595], the gold reward fits

for best-of-$n$ and RL [575]. These $\alpha$ and $\beta$ are fit coef­fi­cients, not the KL coef­fi­cient: $\alpha$ is how well the proxy points uphill, $\beta$ how fast proxy and gold part ways. The best-of-$n$ form held up to $n=\mathrm{60,000}$, about 10 nats [575]. Set­ting each deriv­a­tive to zero gives the peaks:

The paper prints no coef­fi­cients, so we read them off its Figure 3 (dig­i­tized from the paper's figure, approx­i­mate) and fitted the shared ${\alpha }_{\text{RL}}\approx 0.38$ our­selves, assum­ing a nat­u­ral log. The paper's Figure 12 cap­tion calls ${\alpha }_{\text{bon}}/2{\beta }_{\text{bon}}$ the max­i­mum gold score, but it is the peak's loca­tion [575].

Bigger reward models push the peak fur­ther out and higher. And "RL is far less KL-effi­cient than BoN" [575]: RL peaks at roughly 10 to 85 nats and best-of-$n$ at 4 to 14, at sim­i­lar gold scores. A KL number means little with­out its method. In Figure 44, move the size slider and watch the peak mark­ers slide right and up, then com­pare the shaded Llama 3 band near 2 nats with RL's peak far to the right.

What the KL penalty does, and what helps

In Gao et al.'s runs "the effect of the penalty on the gold score is akin to early stop­ping" [575]. Llama 2 found it "useful for train­ing sta­bil­ity, and to reduce reward hack­ing" [229], while Anthropic thought its 0.001 penalty "might actu­ally be wholly unnec­es­sary" [595]. They fit together: the KL term limits which poli­cies are reach­able but is a weak sub­sti­tute for a better reward model. Retrain­ing the reward model on fresh sam­ples over $k$ rounds adds gold reward in pro­por­tion to $\kappa \ln k$ under Gao et al.'s assump­tions [575], though real iter­a­tion dynam­ics remain "poorly under­stood" [597].

Taking the min­i­mum over an ensem­ble of reward models "prac­ti­cally elim­i­nates overop­ti­miza­tion" for best-of-$n$ in a syn­thetic setup with 25% label noise [598]. Under real dis­tri­bu­tion shift, ensem­bles over pre­train­ing seeds beat ones over fine-tuning seeds, but hack­ing per­sists [599]. Aver­ag­ing reward-model weights (WARM) gave a 79.4% win rate over a single reward model [600]. DPO-style meth­ods over-opti­mize too [601] (Sec­tion 8.7).

Length bias

The com­mon­est short­cut is length. Sing­hal et al. found that "even a purely length-based reward repro­duces most down­stream RLHF improve­ments" [602]: length-only PPO's sim­u­lated win rate over the SFT model was 56% against stan­dard PPO's 58% on WebGPT, and 64% against 63% on RLCD. Split­ting each reward gain into within-length-bucket and length-shift parts, they attribute 70% to 90% of it on WebGPT and RLCD to length [602]. Stack Exchange is the excep­tion.

Length-con­trolled AlpacaEval pre­dicts pref­er­ence at zero length dif­fer­ence, which raised its cor­re­la­tion with Chat­bot Arena from 0.94 to 0.98 [603]. ODIN trains a sep­a­rate length head on the reward model and drops it during RL [604].

Syco­phancy

Syco­phancy, telling users what they want to hear, showed up in five AI assis­tants across four tasks. In human pref­er­ence data "match­ing a user’s beliefs, biases, and pref­er­ences is con­sis­tently one of the most pre­dic­tive fea­tures", and a pref­er­ence model favored syco­phan­tic over base­line truth­ful answers to mis­con­cep­tions 95% of the time (reported) [605]. RLHF also raised time-lim­ited eval­u­a­tors' false-pos­i­tive rate by 24.1% on QuALITY and 18.3% on APPS [606].

Watch behav­ior, not reward: KL to the ref­er­ence, length, per­plex­ity, held-out win rates and raw sam­ples. RLHF also "sig­nif­i­cantly reduces output diver­sity com­pared to SFT" [609]. Reward hack­ing in agen­tic envi­ron­ments, which can spread to broader mis­align­ment [610], is in Sec­tion 9.9.

8.7 Direct preference optimization and its variants

Can you reach the RLHF opti­mum from a fixed set of pref­er­ence pairs, with a loss that looks like super­vised learn­ing, and what breaks when you do? Mostly yes. The KL-reg­u­lar­ized objec­tive of Sec­tion 8.5 has a closed-form opti­mum. Read back­wards, every policy defines a reward, and in the Bradley-Terry like­li­hood of Sec­tion 8.4 its nor­mal­izer can­cels. What remains is a logis­tic loss on log-prob­a­bil­ity ratios, with no sam­pling, reward model or value model. This is direct pref­er­ence opti­miza­tion (DPO) [586].

The deriva­tion

Start from ${max}_{\pi }𝔼[r(x,y)]-\beta D_{\text{KL}}(\pi \Vert {\pi }_{\text{ref}})$, where the KL coef­fi­cient $\beta$ con­trols how far the policy may move from the ref­er­ence, usu­ally the SFT model [586]. We assume that this objec­tive is what we want and that pref­er­ences follow Bradley-Terry. Col­lect­ing terms, the objec­tive equals $-\beta {𝔼}_x[D_{\text{KL}}(\pi \Vert {\pi }^{*})-\log Z(x)]$ with

The par­ti­tion func­tion $Z(x)$ does not depend on the policy [586], so if the policy can rep­re­sent any dis­tri­bu­tion, the opti­mum is $\pi ={\pi }^{*}$. On a six-response toy, direct opti­miza­tion of the KL-reg­u­lar­ized objec­tive matches ${\pi }^{*}$ to four dec­i­mal places (com­puted). Take logs and solve for the reward:

Bradley-Terry depends only on reward dif­fer­ences, so $\beta \log Z(x)$ can­cels [586]. Replace ${\pi }^{*}$ by the train­able ${\pi }_{\theta }$ and fit the observed pairs by max­i­mum like­li­hood, with $y_w$ chosen and $y_l$ rejected:

That step assumes finite data lands near the true opti­mum, which is where prac­tice breaks. With the implicit reward ${\widehat{r}}_{\theta }=\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$, the gra­di­ent is

where $g={\nabla }_{\theta }\log {\pi }_{\theta }(y\mid x)$. Each pair pushes the chosen response up and the rejected one down, weighted by how wrongly the implicit reward orders them. With­out the weight the model can degen­er­ate [586]. At step 0 every pair has loss $\ln 2=0.693$ (com­puted), the first number to check in a DPO log.

What $\beta$ means

Write $\Delta$ for the log-ratio margin in nats, so the loss is $-\log \sigma (\beta \Delta )$. Then $\beta$ is a clas­si­fier tem­per­a­ture: an implied pref­er­ence $p$ needs ${\Delta }_p=\ln (p/(1-p))/\beta$, which is 11.0 nats for $p=0.75$ at $\beta =0.1$ and 2.2 at $\beta =0.5$ (com­puted). The gra­di­ent weight $\beta \sigma (-\beta \Delta )$ starts at $\beta /2$, but at a 20-nat margin it is 0.0119 for $\beta =0.1$ and $2.3\times {10}^{-5}$ for $\beta =0.5$ (com­puted). The DPO paper, Zephyr, Llama 3 and KTO default to $\beta =0.1$ [586, 562, 6, 611]. Read $\beta$ in its units. Tülu 3 divides each log-ratio by its length and uses $\beta =5$, which scored 57.3 against 46.8 at $\beta =2$ [550] (reported). That equals a sequence-level $5/|y|$, or 0.017 at 300 tokens (com­puted).

In Figure 45, move $\beta$ and watch how far the loss must travel before it flat­tens, and how the gra­di­ent weight col­lapses once a pair is ordered.

Vari­ants

Read the vari­ants in Table 8.7 as patches to an assump­tion. IPO tar­gets Bradley-Terry: with one label per pair every pref­er­ence looks cer­tain and the KL leash weak­ens [612]. KTO and ORPO cut data or the ref­er­ence, SimPO and length-nor­mal­ized DPO target length. Results con­flict: SimPO reports up to 6.4 points over DPO on AlpacaEval 2 [613] but fell below its start­ing check­point in Tülu 3's abla­tion [550].

R-DPO adds a length-dif­fer­ence term [616], DPOP penal­izes a fall­ing chosen like­li­hood [617], and RPO regresses the implicit gap toward a reward-model gap [291].

What breaks

The loss sees only a dif­fer­ence, so it can fall while both responses become less likely. This like­li­hood dis­place­ment sends mass any­where: train­ing to prefer "No" over "Never" can raise "Yes", and in one exper­i­ment it cut Llama 3 8B Instruct's refusal rate from 74.4% to 33.4% [618] (reported). Llama 3 adds the NLL term and masks shared for­mat­ting tokens, which oth­er­wise caused tail rep­e­ti­tion [6]. Off­line data also lets DPO exploit responses no pair covers [619], and it often degrades before one epoch ends [601], though Zephyr's best model came after three epochs [562]. Pair accu­racy is not a stop­ping rule: judge check­points by held-out gen­er­a­tions, length and refusals.

How to run it. Use the SFT model as the ref­er­ence and cache its log-prob­a­bil­i­ties [550]. Train about one epoch at a learn­ing rate 10 to 100 times below SFT, $2\times {10}^{-7}$ to ${10}^{-6}$ in the DPO paper, Zephyr, Tülu 3 and Qwen2.5 [586, 562, 550, 286]. Llama 3's ${10}^{-5}$, equal to its SFT rate, is an out­lier whose opti­mizer and batch are not stated, so do not trans­fer it [6]. Log the loss, implicit-reward accu­racy, chosen and rejected log-prob­a­bil­i­ties, length and refusals, and eval­u­ate inside the first epoch.

8.8 Online or offline, and how production recipes fit together

Is sam­pling during train­ing a cost to remove or the thing that makes pref­er­ence learn­ing work, and how do real teams com­bine SFT, rejec­tion sam­pling, DPO and RL? Mostly the latter. Keep two axes apart. Off­line or online is when labels are made, before or during train­ing. Off-policy or on-policy is who wrote the responses, another model or the policy itself. DPO on a public dataset is off­line and off-policy, PPO is online and on-policy, and iter­a­tive DPO refreshes at each round. Off­line data is "inevitably off-policy" because the policy moves [620].

At equal KL budget, online losses beat their off­line ver­sions, and scal­ing the policy does not close the gap [621]. With an LLM label­ing the policy's own sam­ples, humans pre­ferred the online ver­sions about 66% of the time [620], and semi-online DPO does about as well as fully online train­ing [622]. The big step is any refresh at all. Whether PPO beats DPO depends on tuning, task and reward (Table 8.8).

The mental model: every method opti­mizes the same objec­tive on paper and dif­fers in where the loss looks. An off­line loss sees only the dataset's responses and says noth­ing about what the policy will write. An online loss sees the policy's own sam­ples and penal­izes their errors, at the price of gen­er­a­tion in the loop. On-policy dis­til­la­tion is the cheap­est such signal: Qwen3 reports it beat RL at a tenth of the GPU-hours [198] (Chap­ter 9).

Sec­tion 8.5 works out PPO's model-state memory: 289 GB for Llama 3 8B and 2,540 GB for 70B. Table 8.9 com­pares it with SFT and DPO. Tülu 3's 8B DPO, 271,409 pairs in 10 hours on 8 H100s [550], implies 6 to 15% MFU for 500 to 1,200 tokens per response (com­puted with $8N$ FLOPs per token, policy train­ing plus ref­er­ence for­ward, against 989.5 TFLOP/s dense BF16).

Pro­duc­tion recipes

Table 8.10 lines up pub­lished recipes, and Figure 41 walks Zephyr, Llama 3 and Tülu 3 stage by stage.

The pat­tern: SFT first, on dis­tilled or rejec­tion-sam­pled data. DPO-family tuning for breadth, on pairs as on-policy as the budget allows. Online RL where rewards are hard to hack, mostly ver­i­fiers (GRPO is derived in Sec­tion 9.3). Then iter­ate, with learn­ing rates fall­ing by stage. Pairs need a large chosen-rejected delta [12], but not the worst sample: $\mu -2\sigma$ of the reward dis­tri­bu­tion beats the min­i­mum [624], which for 100 normal sam­ples is an out­lier at $\mu -2.51\sigma$ (com­puted by Monte Carlo).

8.9 Safety tuning, refusals and character

How do teams make a model refuse what it should, answer what it should, and keep a con­sis­tent char­ac­ter, with the same SFT and pref­er­ence machin­ery? With two error rates. Llama 3 opti­mizes a vio­la­tion rate, responses that break a safety policy, and a false refusal rate, refusals of harm­less prompts [6]. They trade off like pre­ci­sion and recall. Safety train­ing often induces over-refusal, and rea­son­ing before answer­ing eases the trade-off with­out remov­ing it [625].

XSTest mea­sures over-refusal with 250 safe prompts and 200 unsafe con­trasts, built on harm­less uses of alarm­ing words such as killing a Python process, and traces exag­ger­ated safety to "lex­i­cal over­fit­ting" [626]. OR-Bench scales this to 80,000 prompts [627].

The levers are famil­iar. Llama 3 adds bor­der­line prompts, harm­less ones close to harm­ful ones, gives its 8B model a larger share of safety data than 70B, and builds safety DPO pairs "nearly orthog­o­nal in an embed­ding space" [6]. Pref­er­ence tuning can also undo safety, through like­li­hood dis­place­ment (Sec­tion 8.7) or an off-dis­tri­bu­tion ref­er­ence, which left DPO at a 55.4% safety rate [619]. Re-mea­sure both rates after every stage. Table 8.11 lists other tools: red teams find fail­ures, writ­ten rules turn policy into train­ing signal, and a con­strained objec­tive bud­gets harm, as in Safe RLHF:

where $c$ is a learned harm cost and $\lambda$ rises while expected cost exceeds the budget $c_{max}$, like an adap­tive KL coef­fi­cient [628].

Robust­ness mat­ters too. In 13 open chat models, refusal is medi­ated by one direc­tion in acti­va­tion space, which its authors read as brit­tle safety fine-tuning [635].

Char­ac­ter train­ing uses the same tools for pos­i­tive traits. Anthropic added it to Claude 3 through a char­ac­ter vari­ant of Con­sti­tu­tional AI [636]. Open Char­ac­ter Train­ing repro­duces the idea with 11 per­sonas and finds that it "changes manner over con­tent" [637]. Per­sona vec­tors track traits such as syco­phancy and can flag risky train­ing data [638].

You double gra­di­ent-accu­mu­la­tion steps and halve the micro-batch, keep­ing the global batch fixed. Your SFT model now writes worse long answers. What changed?

A per-micro-batch mean weights exam­ples, not tokens: next to a 1,000-token exam­ple, a 10-token one gets 50% of the gra­di­ent instead of 0.99% (com­puted). Nor­mal­ize by the global token count, or use a sum loss with a retuned learn­ing rate as Tülu 3 did [550].

Why can rank-1 LoRA match full fine-tuning for policy-gra­di­ent RL but not for con­tin­ued pre­train­ing on bil­lions of tokens?

A policy gra­di­ent car­ries about one bit per episode, so 10,000 prob­lems with 32 sam­ples each deliver about 320,000 bits, far below rank-1 LoRA's 2.6M param­e­ters on Llama 3 8B (com­puted). Pre­train­ing data car­ries far more infor­ma­tion than an adapter can store [566, 530].

Two label­ers agree on 72.6% of pairs. Your reward model reaches 72% held-out accu­racy. Is a bigger reward model the next step?

Not nec­es­sar­ily. If each labeler flips the truth with prob­a­bil­ity $q$, $(1-q{)}^2+q^2=0.726$ gives $q\approx 0.16$, so even a per­fect model scores only about 84% against single labels (com­puted). Your 72% is well below that cap. Against the true pref­er­ence it is about $(0.72-0.164)/(1-2\times 0.164)\approx 0.83$ (com­puted), so a bigger model still has room: dou­bling reward-model size added about 1.8% accu­racy in Sti­en­non et al.'s runs [572]. First rela­bel a held-out set with sev­eral labels per pair, so you can mea­sure which error dom­i­nates [8].

A PPO run shows rising reward, smooth losses, longer responses and fall­ing per­plex­ity. What is hap­pen­ing, and what do you check?

Likely reward hack­ing toward length, which reward and loss curves cannot reveal [596, 602]. Check KL, reward within length buck­ets, held-out win rates and raw sam­ples.

Tülu 3 uses $\beta =5$ and Llama 3 uses $\beta =0.1$. Which keeps the policy closer to its ref­er­ence?

The num­bers alone cannot say. Tülu 3's $\beta$ is per token, so at 300 tokens it equals $5/300=0.017$ per sequence, below 0.1 (com­puted). Learn­ing rates (${10}^{-5}$ against $5\times {10}^{-7}$) also decide how far the policy moves [550, 6].

9. Post-training II: reinforcement learning for reasoning and agents

A check­able answer sup­plies a reward, but a reward alone does not spec­ify a sound learn­ing update. Follow it through the base­line, impor­tance weights and policy loss, then account for the sam­pling system and the short­cuts the ver­i­fier leaves open.

DeepSeek-R1-Zero started from DeepSeek-V3-Base, a mix­ture-of-experts model with 671B total and 37B active param­e­ters [2], with no super­vised fine-tuning. GRPO (Group Rel­a­tive Policy Opti­miza­tion, Sec­tion 9.3) trained it on two rule-based rewards, one for a cor­rect answer and one for the required format, "com­bined with the same weight" [3]. Its aver­age pass@1 on AIME 2024 (the chance that one sam­pled answer is right) rose from 15.6% to 77.9% [3, 639] (reported) for about 3.6% of DeepSeek-V3's full train­ing GPU-hours (com­puted, Sec­tion 9.4 has the recipe and the cost).

In Chap­ter 8 the reward was a learned pref­er­ence model, so the policy had to stay near that model's train­ing data. Here the reward is a checker: an answer matcher, a test run, a format rule. Its ver­dict does not degrade as the policy drifts, so train­ing can run for thou­sands of steps and answers can reach 65,536 tokens [3]. That free­dom brings three prob­lems: esti­ma­tor details (Sec­tions 9.2 to 9.6), the cost of long gen­er­a­tion (Sec­tion 9.8) and poli­cies that sat­isfy the checker with­out doing the task (Sec­tion 9.9).

Other run­ning exam­ples are Tülu 3, which named the method and ran it on Llama 3.1 8B to 405B [550], DeepSWE, a Qwen3⁠–⁠32B coding agent trained only with RL [9], and Llama 3 8B and 70B for memory arith­metic [6]. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. Here $G$ is the group size, $p$ a prompt's pass rate and $\rho$ a token's prob­a­bil­ity ratio. See also Chap­ter 6 (RL scal­ing laws), Chap­ter 13 (decod­ing) and Chap­ter 14 (sand­boxes).

9.1 Rewards you can check

What number does RL opti­mize when the task has a right answer, and how does that number go wrong? RL opti­mizes what the reward mea­sures, not what you meant. With a ver­i­fi­able reward the mea­sure­ment is a pro­gram, so the first engi­neer­ing job is to make it agree with a care­ful human grader.

Tülu 3 defined rein­force­ment learn­ing with ver­i­fi­able rewards (RLVR) as a method that "lever­ages the exist­ing RLHF objec­tive but replaces the reward model with a ver­i­fi­ca­tion func­tion" [550]. A ver­i­fied com­ple­tion earns a con­stant $\alpha =10$, and PPO opti­mizes it with a KL coef­fi­cient of 0.05 at 8B and 0.07 at 70B on roughly 30,000 prompts from GSM8K, MATH and instruc­tion-fol­low­ing con­straints [550] (reported). R1-Zero's accu­racy reward matches a boxed answer or runs code against tests, and its format reward checks for think tags. DeepSeek avoided neural reward models because they get hacked in large-scale RL [3]. DAPO, an open 2025 recipe built on GRPO (Sec­tion 9.5), had an LLM rewrite each prob­lem so the answer is an inte­ger, $a+b+c$ for $(a+b^{1/2})/c$, which makes check­ing an inte­ger com­par­i­son [640].

Learned ver­i­fiers

Before RLVR, the field trained models to judge solu­tions. An out­come reward model (ORM) scores a whole solu­tion, a process reward model (PRM) each step (Table 9.1). Light­man et al.'s 500 held-out test prob­lems became MATH-500 [642].

What does a PRM esti­mate? For steps $s_1,\dots ,s_T$, the value of a prefix is the chance that the final answer is cor­rect, and a step's advan­tage is the change in that value:

Math-Shep­herd's soft label, the share of roll­outs from a prefix that suc­ceed, is a Monte Carlo esti­mate of the value $V^{\pi }$ [644]. Setlur et al.'s progress reward is the advan­tage $A^{\mu }$ under a fixed prover $\mu$ [645]. Human labels mea­sure valid­ity, and a valid but use­less step makes zero progress.

So fron­tier recipes went back to out­come rewards. DeepSeek wrote that "once a model-based PRM is intro­duced, it inevitably leads to reward hack­ing" [3].

Ver­i­fi­able rewards still fail. False neg­a­tives teach the policy to avoid unusual cor­rect forms. False pos­i­tives invite hack­ing (Sec­tion 9.9). Trun­ca­tion scores a cut-off answer as 0: mask­ing trun­cated sam­ples was worth 6 AIME points in DAPO's abla­tion [640], and ScaleRL, a study of RL recipe choices that spent more than 400,000 GPU-hours (Sec­tion 9.5), saw 10⁠–⁠15% trun­ca­tion desta­bi­lize train­ing, so it forces an end of think­ing with "Okay, time is up. Let me stop think­ing and for­mu­late a final answer now. </think>" [483]. A format reward can be col­lected with­out solv­ing any­thing: reward­ing any non-empty boxed answer lifted Qwen2.5-Math-7B by 10.3 points on AIME 2024, against 15.3 for the true reward [649] (reported, case study in Sec­tion 9.7).

Prac­ti­cal defaults, as of Novem­ber 2026: canon­i­cal answers, one checker for train­ing and eval­u­a­tion, masked trun­ca­tion, a small format reward, and a dis­agree­ment rate mea­sured on a few hun­dred hand-graded sam­ples and logged during the run. Keep PRMs for test-time rerank­ing and search [71, 645]. Grader sta­tis­tics are in Chap­ter 10.

9.2 From a score to a gradient: REINFORCE, baselines and PPO

Sam­pling a token is a dis­crete choice, so how do we get a gra­di­ent that raises a 0/1 score, and how far can we move on one batch of sam­ples? Make answers more or less likely in pro­por­tion to how much better or worse than expected they scored. That is REINFORCE, and later meth­ods change only its mul­ti­plier, its base­line or its step size.

The score-func­tion esti­ma­tor

A prompt $x$ yields a response $y=(y_1,\dots ,y_T)$ with prob­a­bil­ity ${\pi }_{\theta }(y\mid x)$, and we want to raise $J(\theta )={𝔼}_{y~{\pi }_{\theta }}[R(x,y)]$. Write the expec­ta­tion as a sum and use $\nabla \pi =\pi \nabla \log \pi$:

The reward is never dif­fer­en­ti­ated, so it can be a test suite, and one for­ward and back­ward pass over the sam­pled response, fed in as fixed input (teacher forc­ing), gives the gra­di­ent. Williams named the family: REINFORCE algo­rithms "make weight adjust­ments in a direc­tion that lies along the gra­di­ent of expected rein­force­ment" [650].

Sub­tract a base­line $b$ that depends on the prompt but not on the sam­pled answer:

The mean is unchanged and, for a good $b$, the vari­ance falls. The nat­u­ral choice is the expected reward $V(x)$, and $R-V(x)$ is the advan­tage. On a toy five-way soft­max (logits 1, 0.5, 0, -0.5, -1, the first and fourth answers rewarded), 200,000 sam­ples match the exact gra­di­ent either way, and the base­line halves the vari­ance summed over the five logits, 0.22 to 0.11 (com­puted, illus­tra­tive). The base­line must be per prompt: within one Llama-2⁠–⁠7B mini-batch, ReMax saw learned rewards from -14.25 to 7.25 [651].

Only the end of a response is scored, so Ahma­dian et al. argue "it may be more appro­pri­ate to model the entire gen­er­a­tion as a single action, as opposed to each token" [652]. This bandit view is why critic-free meth­ods took over.

With $R\in \{0,1\}$ and $b=0$ the gra­di­ent is max­i­mum like­li­hood on the suc­cesses, which is filter-and-fine-tune. STaR [653] and ReST-EM [654] work this way, and ReST-EM's gains fade after a couple of iter­a­tions. What is miss­ing is the push down on wrong answers: Kimi k1.5 found neg­a­tive gra­di­ents markedly help on long chains of thought [648], and a small 2026 pre­print found GRPO beat rejec­tion-sam­pling fine-tuning by 10.9 points [655] (reported, pre­print).

PPO, and what long answers do to its critic

PPO as used in RLHF (clipped sur­ro­gate, GAE, value model) is derived in Sec­tion 8.5. We restate two for­mu­las and ask what long answers change. PPO takes sev­eral steps per batch sam­pled by ${\pi }_{{\theta }_{\text{old}}}$ and clips each token's ratio, by default at $ϵ=0.2$ [591]:

The advan­tage comes from gen­er­al­ized advan­tage esti­ma­tion (GAE) over a learned value model [592]:

With one reward at the end and $\gamma =1$, a token $d$ posi­tions ear­lier sees that reward with weight ${\lambda }^d$. At the common $\lambda =0.95$ that is about 0.006 after 100 tokens, "effec­tively zero" as VAPO puts it [657], and $5\times {10}^{-23}$ after 1,000 (com­puted). With $\gamma =\lambda =1$ the sum tele­scopes to $R-V_{\psi }(x,y_{<t})$, and Open-Rea­soner-Zero scaled vanilla PPO this way with no KL term [658].

The critic is typ­i­cally as large as the policy [207], and its train­ing state took 46% of GPU memory at 7B [651]. Ini­tial­ized from a reward model, as usual [550], vanilla PPO from Qwen2.5⁠–⁠32B base "only achieves 5 points" on AIME 2024 [657]. In DeepSeek's test on a 16B MoE model, PPO at $\lambda =0.95$ did con­sid­er­ably worse than GRPO and at $\lambda =1.0$ came close [3].

Sec­tion 8.5 works out PPO's model-state memory: 289 GB for Llama 3 8B and 2,540 GB for 70B. GRPO with a rule reward trains only the policy (16 bytes per param­e­ter) and keeps a frozen BF16 ref­er­ence (2 bytes), 18 bytes against PPO's 36: 145 GB and 1,270 GB, or 2 and 16 GPUs of 80 GB instead of 4 and 32 (com­puted from the param­e­ter counts [6]), before acti­va­tions.

If roll­out and learner log-prob­a­bil­i­ties differ, as with dropout, "all ratios could poten­tially be clipped, result­ing in zero gra­di­ent" [550]. Such code-level details can matter as much as the algo­rithm [593, 659, 660]. Even in RLHF, REINFORCE-style meth­ods beat PPO by 3.2% to 20.3% in win rate, and remov­ing clip­ping slightly helped [652] (Table 9.2).

Prac­ti­cal defaults: a per-prompt base­line from sev­eral sam­ples, never a raw 0/1 reward. If you keep a critic, use $\lambda =1$ and pre­train it (Sec­tion 9.5). Dis­able dropout and recom­pute old log-prob­a­bil­i­ties in the trainer (Sec­tion 9.6). Byte account­ing is in Chap­ter 1.

9.3 Dropping the critic: group baselines and their biases

With­out a critic, where does the base­line come from, and what do the pop­u­lar choices do to each token's gra­di­ent? Sample sev­eral answers to the same prompt. Their aver­age score esti­mates how hard the prompt is for the cur­rent policy, which is the base­line we want. The vari­ants differ only in scale.

GRPO, Group Rel­a­tive Policy Opti­miza­tion, "fore­goes the critic model, instead esti­mat­ing the base­line from group scores" [207]. Each token of answer $o_i$, with length $|o_i|$ and reward $r_i$, gets the group z-score, and the loss aver­ages clipped token terms ${\ell }_{i,t}$ within each answer, then over the group:

Here $\beta$ is the KL coef­fi­cient, and the KL sits in the loss, not the reward [207] (Sec­tion 9.6). RLOO, REINFORCE leave-one-out, uses the mean of the other answers [652], and since $R_i-(G\overline{R}-R_i)/(G-1)=G(R_i-\overline{R})/(G-1)$,

All fit one tem­plate, with a nor­mal­izer $Z$ ($G|o_i|$ in GRPO, total tokens in DAPO, a con­stant in Dr. GRPO) and a per-token weight $w$ that Sec­tion 9.5 com­pares:

With $k$ of $G$ cor­rect and $\widehat{p}=k/G$, GRPO gives cor­rect answers $+[(1-\widehat{p})/\widehat{p}{]}^{1/2}$ and wrong ones $-[\widehat{p}/(1-\widehat{p}){]}^{1/2}$. Every esti­ma­tor sums to zero over the group, moving prob­a­bil­ity from fail­ures to suc­cesses.

Dr. GRPO: two biases in the objec­tive

Liu et al. found that GRPO's $1/|o_i|$ term "arti­fi­cially increases response length (espe­cially for incor­rect out­puts) during train­ing", and that its std over­weights prompts that are too easy or too hard [662]. Remov­ing both and divid­ing by a con­stant recov­ers the PPO objec­tive with a Monte Carlo advan­tage and an unbi­ased base­line [662].

Why length? A neg­a­tive advan­tage is a fixed push per answer, spread over its tokens. Take $G=4$, rewards (1, 1, 0, 0), lengths (100, 100, 100, 1,000). GRPO's advan­tages are $\pm 1$, so its per-token weight is $-1/(4\times 100)=-0.0025$ on the short wrong answer and $-0.00025$ on the long one. DAPO's token mean gives $-1/\mathrm{1,300}$ to both, so the long answer takes ten times the total push. Dr. GRPO with con­stant $4\times \mathrm{4,096}$ gives $-3.05\times {10}^{-5}$ to both, inde­pen­dent of the other answers (com­puted).

Why dif­fi­culty? Summed over a group with pass rate $p$,

The ratio is 2.0 at $p=0.5$, 3.0 at $1/8$ and 4.1 at $1/16$ (com­puted), so a one-in-six­teen prompt gets twice the rel­a­tive weight of a coin flip. Open-source PPO code also nor­mal­ized by length [662], and verl now makes it a choice, loss_agg_mode [663]. In Figure 46, click "Per-token weight", then move the length slid­ers and watch the three aggre­ga­tions diverge.

Zero-vari­ance groups

If all $G$ answers get the same reward, every advan­tage is zero, which shrinks and adds noise to the batch gra­di­ent [640], yet the roll­outs still cost $G$ gen­er­a­tions. With inde­pen­dent sam­ples at pass rate $p$ (Table 9.5),

As the policy improves, the waste grows. DAPO's dynamic sam­pling over-sam­ples and dis­cards these prompts [640] at a cost of

1.29 at $G=8$ for uni­form pass rates and 1.50 for Beta(4,1) (com­puted). For a DAPO-style batch of 512 prompts by 16 answers, from a pool half nearly mas­tered ($p=0.95$, $P_0=0.44$) and half at $p=0.3$ ($P_0=0.003$), $𝔼[P_0]=0.22$ and each step draws $512/0.78\approx 660$ prompts, about 10,500 roll­outs (com­puted, illus­tra­tive pool). Kept in the batch, zero-advan­tage groups still feed the KL and entropy terms and raise their rel­a­tive weight, which can desta­bi­lize train­ing, so Sky­work-OR1 drops them [664]. Figure 46 shows both effects. On the left, mark every answer cor­rect and all bars drop to zero. On the right, move the pass rate and watch $P_0$ climb at both ends for every $G$.

Larger groups cost more gen­er­a­tion. At $p=0.05$ the chance of at least one suc­cess, $1-{0.95}^G$, is 34% at $G=8$, 56% at 16 and 96% at 64 (com­puted). DeepSeekMath used 64 sam­ples of up to 1,024 tokens [207], and R1-Zero, whose answers run far longer, used 16 [3].

Fil­ter­ing is most of the trick. GRPO's main advan­tage comes from dis­card­ing all-wrong prompts, not from its nor­mal­iza­tion [665]. Sky­work-OR1 removes prompts the base model always or never solves [664], and ScaleRL drops those with pass rate 0.9 or more [483]. A 2026 pre­print finds roughly 88% of a pooled open math corpus solved or failed in every roll­out [666] (reported, pre­print), so fil­ter­ing even­tu­ally runs out of prob­lems.

Should you divide by the std? GRPO uses the group's [207], Dr. GRPO none [662], REINFORCE++ and "Lite PPO" the batch's [661, 667]. ScaleRL found the choices sim­i­lar [483], and Magis­tral saw no sig­nif­i­cant effect [668]. Our read­ing: it moves effi­ciency and sta­bil­ity more than the final score, and GRPO's std acts as a cur­ricu­lum toward hard-but-solv­able prompts.

Prac­ti­cal defaults, as of Novem­ber 2026: $G$ of 8 to 16, group-mean cen­ter­ing, no std or a batch std, and zero-vari­ance groups dropped and logged every step, since their share reads out the cur­ricu­lum.

9.4 DeepSeek-R1: reasoning learned by reinforcement

What hap­pens when GRPO with rule-based rewards runs on a very strong base model, and how do you turn the result into a model people can use? The model learns to think at length before it answers, and its com­pe­ti­tion-math score rises five­fold. Making it usable takes three more stages.

A trans­former spends fixed com­pu­ta­tion per token, and writ­ing inter­me­di­ate steps into its con­text buys more serial com­pu­ta­tion and a scratch­pad [669]. Chain-of-thought prompt­ing took PaLM 540B from 17.9% to 56.9% on GSM8K, a gain that appeared only at about 100B param­e­ters [670]. RL with a checker rewards what­ever writ­ing leads to right answers, so rea­son­ing the base model already pro­duces now and then grows longer and more fre­quent. For o1, OpenAI pub­lished only that it "is trained with large-scale rein­force­ment learn­ing to reason using chain of thought" [671]. The most detailed public recipe is DeepSeek-R1 [3, 639].

R1-Zero: RL on a base model and noth­ing else

R1-Zero runs GRPO (Sec­tion 9.3) on DeepSeek-V3-Base with two rule-based rewards, answer accu­racy and format, and a tem­plate that only asks for rea­son­ing inside <think> tags. Table 9.6 lists the set­tings.

During RL, aver­age pass@1 on AIME 2024 rose from 15.6% to 77.9%, and self-con­sis­tency over 16 sam­ples (cons@16) reached 86.7% [3, 639]. Accu­racy and length both jump at step 8.2k [3], where the cap dou­bled, so the cap had bound. An "aha moment" appeared, "char­ac­ter­ized by a sudden increase in the use of the word “wait” during reflec­tions", but the out­puts were hard to read and mixed Eng­lish and Chi­nese [3]. The first arXiv ver­sion (Jan­u­ary 2025) reported 71.0% pass@1 and 86.7% cons@64 [3] (reported). These are two reports of one run, so use the later num­bers.

The base decides whether any of this hap­pens. DeepSeek's 7B dense and 16B mix­ture-of-experts bases "con­sis­tently failed to yield mean­ing­ful improve­ments" on AIME and drifted into rep­e­ti­tion, while bases of 32B, 230B and 671B worked [3].

From R1-Zero to R1: four stages

1. Cold start: SFT on thou­sands of read­able long rea­son­ing exam­ples.
2. Rea­son­ing RL as in R1-Zero, plus a lan­guage-con­sis­tency reward, "the pro­por­tion of target lan­guage words in the CoT" (chain of thought), and a GRPO clip ratio of 10 [3], which barely clips and so gives up most of PPO's step-size con­trol (Table 9.9 fol­lows one rare token through it).
3. Rejec­tion sam­pling and SFT on about 600k rea­son­ing and 200k other sam­ples [3].
4. Second RL stage: 1,700 steps, with pref­er­ence rewards only in the last 400 [3].

AIME 2024 pass@1 goes 77.9, 59.0, 74.0, 78.1 and 79.8 across the five check­points (the dip comes from the small cold-start set). IF-Eval climbs from 46.6 to 83.3 and AlpacaEval 2.0 from 24.7 to 87.6. GPQA Dia­mond falls from 75.8 to 71.5 [3] (reported). One AIME prob­lem is 3.3 points, so the last stage's 78.1 to 79.8 is half a prob­lem (com­puted, Sec­tion 10.4). Figure 47 plots ten bench­marks per stage: look for the red bracket on GPQA, where R1 ends below R1-Zero, and for dis­til­la­tion beat­ing RL in the "Dis­til­la­tion vs RL" tab.

Process rewards and tree search did not work for DeepSeek, because text has "an expo­nen­tially larger search space" than board games [3].

Dis­till small models, train large ones

R1's small models get SFT only. From the same Qwen2.5⁠–⁠32B base, more than 10K steps of RL reached 47.0% on AIME 2024, and SFT on R1's out­puts 72.6% [3]. Qwen3 found the same with on-policy dis­til­la­tion (the stu­dent sam­ples, the teacher's logits grade every token). From one dis­tilled 8B check­point at 55.0%, GRPO reached 67.6% in 17,920 GPU-hours and dis­til­la­tion 74.4% in 1,800, and only dis­til­la­tion raised pass@64, from 90.0% to 93.3% [198] (reported). The cost and scal­ing law of dis­til­la­tion are in Sec­tion 7.7.

Is the aha moment emer­gent, as DeepSeek presents it? The SimpleRL-Zoo and cog­ni­tive-behav­ior results sup­port a nar­rower read­ing, better sup­ported as of Novem­ber 2026: RL ampli­fies a behav­ior the base already shows rarely. How test-time and RL com­pute scale is in Sec­tion 6.8.

To do it your­self, start from the strong­est base you can afford, use rule-based rewards, keep prompts the model some­times solves, sample 8 to 16 answers per prompt and raise the length cap in stages. For small models, dis­till first.

9.5 What the reproductions exposed: DAPO, VAPO, GSPO and CISPO

DAPO's open rerun of R1-Zero's recipe on Qwen2.5⁠–⁠32B scored 30 on AIME instead of DeepSeek's 47. What did each 2025 fix change in the loss, and which ones change the final score rather than the speed? Each fix turns one knob of the tem­plate in Sec­tion 9.3: the clip range, which groups enter the batch, how token losses are aver­aged, the level of the prob­a­bil­ity ratio, or whether a token's gra­di­ent can be zeroed. Here ${\rho }_{i,t}={\pi }_{\theta }/{\pi }_{{\theta }_{\text{old}}}$ is the token prob­a­bil­ity ratio.

DAPO: four fixes and an abla­tion

"In our ini­tial GRPO run, we achieved only 30 points on AIME", the DAPO team wrote, against DeepSeek's 47 from the same base. DAPO drops the KL term ("this restric­tion is not nec­es­sary") and adds four tech­niques [640].

• Clip-higher. With a pos­i­tive advan­tage, clip­ping zeroes a token's gra­di­ent once its prob­a­bil­ity passes
  $${\pi }_{max}={\pi }_{\text{old}}(1+{ϵ}_{\text{high}}).$$
  At $ϵ=0.2$ a token at 0.01 can reach only 0.012, while one at 0.9 is uncon­strained. DAPO raises the upper bound to 0.28, so the rare token reaches 0.0128 (com­puted), and keeps the lower bound at 0.2 so unlikely tokens are not pushed to zero [640]. The target is entropy col­lapse: the policy's token dis­tri­bu­tion nar­rows until every answer in a group agrees, and learn­ing stalls (Sec­tion 9.6).
• Dynamic sam­pling. Over-sample and drop groups whose answers all pass or all fail. It is cheap in wall-clock time, since "the gen­er­a­tion time is typ­i­cally dom­i­nated by the gen­er­a­tion of long-tail sam­ples" [640].
• Token-level loss. Aver­age over every token in the group, so a long answer counts in pro­por­tion to its length and long gib­ber­ish is penal­ized token by token [640].
• Over­long shap­ing. Mask the loss of trun­cated answers, then penal­ize length lin­early over the last $L_{\text{cache}}$ tokens before the cap [640]:
  $$R_{\text{length}}(y)=\{\begin{array}{cc}0,\hfill & |y|\le L_{max}-L_{\text{cache}}\hfill \\ {\displaystyle \frac{(L_{max}-L_{\text{cache}})-|y|}{L_{\text{cache}}}},\hfill & L_{max}-L_{\text{cache}}<|y|\le L_{max}\hfill \\ -1,\hfill & |y|>L_{max}\hfill \end{array}$$
  With $L_{max}=\mathrm{20,480}$ and $L_{\text{cache}}=\mathrm{4,096}$, an 18,000-token answer gets $(\mathrm{16,384}-\mathrm{18,000})/\mathrm{4,096}=-0.39$ and a 20,000-token answer $-0.88$ (com­puted).

Read Table 9.8 with care. DAPO's incre­ments depend on the order, there are no error bars, and one AIME prob­lem is 3.3 points (com­puted), so the +1 and +2 are within noise while the +6 and +8 are not. The ref­er­ence 47 was sam­pled at tem­per­a­ture 0.6 and top-p 0.95 [3], a dif­fer­ent pro­to­col.

VAPO: keep the critic and repair it

VAPO keeps PPO's value model and adds seven changes, among them value pre­train­ing and a length-adap­tive GAE with ${\lambda }_{\text{policy}}=1-1/(\alpha l)$, $\alpha =0.05$ [657]. Since $1/(1-\lambda )=\alpha l$, a 1,000-token answer gets $\lambda =0.98$, a 50-token hori­zon, and a 16,000-token answer 0.99875, an 800-token hori­zon (com­puted). VAPO reaches 60.4 "within a mere 5,000 steps", 60 to 61 across three runs [657] (reported).

GSPO: one ratio per response

Qwen's GSPO argues that a token ratio is an impor­tance weight from one sample of each next-token dis­tri­bu­tion, noise that grows with length, and that "the unit of opti­miza­tion objec­tive should match the unit of reward". It clips one length-nor­mal­ized ratio per response, at 3e-4 below and 4e-4 above [678]:

For mix­ture-of-experts models, about 10% of the experts Qwen3⁠–⁠30B-A3B acti­vates change after each update. GRPO needed rout­ing replay, which reuses the old policy's expert choices, to con­verge, and GSPO did not [678].

CISPO: clip the weight, never drop the token

In MiniMax-M1, rare reflec­tive tokens such as "Wait" got large ratios and "were clipped out after the first on-policy update", so they missed the remain­ing off-policy updates. CISPO caps a stop-gra­di­ent impor­tance weight instead, with no lower bound [656]:

Every token keeps a gra­di­ent of capped size. MiniMax reports a 2x speedup over DAPO, and a full RL run of three weeks on 512 H800 GPUs for $534,700 [656], which is 258,048 GPU-hours at about $2.07 each (com­puted). Table 9.9 fol­lows one "Wait" token through each method.

What the head-to-head stud­ies found

ScaleRL spent more than 400,000 GPU-hours asking which choices raise the ceil­ing of a fitted reward-versus-com­pute curve (fit in Sec­tion 6.8). CISPO and GSPO raised it over DAPO, and an FP32 lan­guage-model head lifted it from 0.52 to 0.61, with runs vary­ing by about 0.02. Most other choices changed only effi­ciency [483] (reported). Table 9.10 lines up the algo­rithms.

Four dis­putes have work­ing res­o­lu­tions. Aggre­gate per token from a base model and per sequence from an aligned one [667]. A repaired critic beat DAPO by 10 points (Table 9.8), though group meth­ods are sim­pler and halve model memory. GRPO with roll­out rout­ing replay beat GSPO by 1.29 on an MoE model [679], so fixing noisy ratios at their source (Sec­tion 9.6) recov­ers most of what sequence ratios buy. And clip-higher "grad­u­ally becomes unsta­ble" next to covari­ance-based meth­ods [680], so tune the clip as an entropy lever.

9.6 Keeping a run alive: KL, entropy and the sampler-trainer mismatch

Why do RL runs plateau or col­lapse, and which quan­ti­ties do you mon­i­tor to keep them learn­ing? Three things qui­etly break the gra­di­ent: a KL term that pulls in an unin­tended direc­tion, a policy that stops explor­ing so every group agrees, and sam­ples from a slightly dif­fer­ent policy than the one trained, a bias that grows with length.

KL: keep it, drop it or reset it

RLHF needs a KL term because a learned reward is accu­rate only near its train­ing data. Rea­son­ing RL "typ­i­cally employs rule-based ver­i­fiers", which removes that con­cern, so Dr. GRPO sets the coef­fi­cient to zero [662]. Drop­ping it saves a ref­er­ence for­ward pass per step (Table 9.11).

KL also costs breadth: a 0.001 penalty matched pass@1 but gave "a much lower pass@128" [484]. So drop the term from a base model with a rule-based reward, and keep a small one with peri­odic resets for long runs from SFT or dis­tilled models.

Which gra­di­ent a KL esti­ma­tor gives

Schul­man's single-sample esti­ma­tors, for a sam­pled token $y~{\pi }_{\theta }$, are [590]

GRPO adds $k_3$ to its loss [207]. Its value is an unbi­ased esti­mate of the reverse KL, $\mathrm{K}\mathrm{L}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$. Its gra­di­ent is another matter. Since $\nabla \delta =-\delta \nabla \log {\pi }_{\theta }$ and $𝔼[\nabla \log {\pi }_{\theta }]=0$,

the gra­di­ent of the for­ward KL. The same steps give the reverse-KL gra­di­ent for $k_2$ and zero for $k_1$ (com­puted).

Entropy col­lapse

In entropy col­lapse, entropy falls early, groups become iden­ti­cal and learn­ing plateaus. In one run "73% of the entropy con­sump­tion and 76% of the per­for­mance gain occurred in just the first 200 gra­di­ent steps". Cui et al. fit $R=-a\exp (\mathcal{H})+b$ between entropy and val­i­da­tion score, with a ceil­ing of $b-a$ at zero entropy, and early-step fits pre­dicted later scores to about 1%. The mech­a­nism is a covari­ance [680]:

Under nat­u­ral policy gra­di­ent the logit change is $\eta$ times the advan­tage. When the likely tokens are also the rewarded ones, the covari­ance is pos­i­tive and entropy falls. On a 1,000-action soft­max with $\eta =0.01$, the for­mula pre­dicts $-0.0264$ and the exact change is $-0.0264$ (com­puted).

Covari­ance-tar­geted clip­ping beat GRPO by 2.0% at 7B and 6.4% at 32B [680]. Train­ing only on the 20% high­est-entropy tokens beat full updates on Qwen3⁠–⁠32B by 11.04 points on AIME'25 [682]. Higher tem­per­a­ture "only delays the onset of entropy col­lapse" [485], and on-policy updates slow it [664]. Col­lapse also shows inside one sample: MiniMax halts gen­er­a­tion once 3,000 con­sec­u­tive tokens each have prob­a­bil­ity above 0.99, the sig­na­ture of a rep­e­ti­tion loop [656].

The sam­pler is not the trainer

Sam­ples go stale: R1-Zero's mean mini­batch is 7.5 updates old [3] (com­puted), and Magis­tral degrades beyond 2 mini­batches per batch [668] (sys­tems in Sec­tion 9.8). And even with iden­ti­cal weights, sam­pler and trainer run dif­fer­ent ker­nels, so ${\pi }_{\text{infer}}\ne {\pi }_{\text{train}}$, and "the policy gra­di­ent would become biased if simply ignor­ing this mis­match" [683]. The exact fix is a whole-response weight $w={\prod }_t{\rho }_t$ with ${\rho }_t={\pi }_{\text{train}}/{\pi }_{\text{infer}}$. Prac­ti­cal fixes trun­cate or mask it [683], or trun­cate per token [684, 685].

Why not use $w$ itself? Model each token's log-ratio as inde­pen­dent and normal with spread $\sigma$ and mean $-{\sigma }^2/2$, so each token's weight aver­ages 1. Over $T$ tokens

where ESS is the nor­mal­ized effec­tive sample size, $({\sum }_i{w}_i{)}^2/(N{\sum }_i{w}_i^2)$, exactly 1 on-policy [686]. With $\sigma =0.03$ (illus­tra­tive), the vari­ance is 0.094 at 100 tokens, 1.46 at 1,000 and about 1,300 at 8,000. At 8,000 tokens the median weight is 0.027, and cap­ping at 2 cuts the mean weight to 0.25, a 75% down­ward bias (com­puted). Figure 48 lets you move $\sigma$, $T$ and the cap. Watch the effec­tive sample size halve within the first thou­sand tokens and, for the dense preset, fall below 0.1 by about 1,800 (com­puted).

Ma et al. esti­mate train-infer­ence KL per token with $k_3$, and mea­sured $6.4\times {10}^{-4}$ for dense Qwen3⁠–⁠8B, $1.535\times {10}^{-3}$ for MoE Qwen3⁠–⁠30B-A3B and $7.5\times {10}^{-4}$ after rout­ing replay [679]. Under the model $𝔼[k_3]={\sigma }^2/2$, so $\sigma =\sqrt{2\mathrm{K}\mathrm{L}}\approx 0.036$, 0.055 and 0.039, and the sequence ESS $e^{-2\mathrm{K}\mathrm{L}T}$ is 0.28, 0.05 and 0.22 at 1,000 tokens (com­puted). Real mis­match is heavy-tailed and cor­re­lated [679], so the model only teaches.

Better to attack the cause, as in the case study below. FP16 has 8 times BF16's pre­ci­sion (Sec­tion 5.6), and the FP16 paper prices impor­tance-sam­pling patches at one extra for­ward pass, about 25% of train­ing cost [683]: for­ward plus back­ward is 3 units and one more for­ward makes 4, so the extra pass is a quar­ter of the new step, or a third more than before (com­puted). In MoE models about 10% of router deci­sions and 94% of tokens pick dif­fer­ent experts in the two engines. Roll­out Rout­ing Replay (R3) reuses the sam­pler's rout­ing in train­ing [679] (Sec­tion 4.7), unlike GSPO's replay inside the trainer. Unrepli­cated 2026 pre­prints split the error into a fixed numeric part and a grow­ing policy lag [687, 688, 689, 690].

What to log

Recom­pute old log-prob­a­bil­i­ties in the trainer [678]. Each step, log entropy, length, the zero-vari­ance frac­tion, train-infer­ence KL, the share of ratios above 2 and the clip frac­tion. Keep the LM head in FP32 and stal­e­ness to a few updates. What to mon­i­tor during a run is in Sec­tion 12.5. Why iden­ti­cal prompts at tem­per­a­ture 0 give dif­fer­ent out­puts (80 com­ple­tions in Think­ing Machines' test) is in Sec­tion 12.7.

9.7 Thinking longer, and what RL actually adds

How should infer­ence com­pute be spent on a trained model, and does RL with ver­i­fi­able rewards teach new rea­son­ing or only make exist­ing rea­son­ing more likely? You can sample many attempts and aggre­gate them (par­al­lel com­pute) or make one attempt longer (sequen­tial com­pute). Rea­son­ing models are trained for the second. The first gives a mea­sur­ing stick: pass@k at large $k$ counts a prob­lem as solved if any attempt works, so it mea­sures what a model can ever solve. How both scale is in Sec­tion 6.8.

Par­al­lel com­pute: voting is not pass@k

Self-con­sis­tency sam­ples sev­eral rea­son­ing paths and keeps the most fre­quent answer, which raised GSM8K accu­racy by 17.9 points [691]. Cov­er­age, the share of prob­lems solved by any of $k$ sam­ples, keeps rising over four orders of mag­ni­tude of $k$ [479]. A major­ity vote (cons@k) needs no oracle but con­verges to the answer the model prefers: with the right answer at 0.30 and one wrong answer at 0.45, pass@64 is 1.00 and cons@64 about 0.08 (com­puted, illus­tra­tive). DeepSeek-R1 scores 79.8% pass@1, 86.7% cons@64 and 90.0% pass@64 on AIME 2024, while voting lifts GPT-4o only from 9.3% to 13.4% [3] (reported).

Sequen­tial com­pute: budget forc­ing and think­ing bud­gets

s1 fine-tuned Qwen2.5⁠–⁠32B-Instruct on 1,000 selected ques­tions in 26 min­utes on 16 H100s. At decode time, budget forc­ing caps think­ing by append­ing the end-of-think­ing delim­iter, and extends it by sup­press­ing the delim­iter and append­ing "Wait" [482]. Of the meth­ods in Table 9.12 it alone both respects the budget and gains from more tokens, and pair­ing the others with it scores lower [482]. It takes AIME 2024 from 50% to 57%. That is two prob­lems out of 30, while a single-sample score at accu­racy 0.5 has a stan­dard error of $\sqrt{0.25/30}$, 9.1 points (com­puted). Error bars are in Sec­tion 10.4.

Prod­ucts expose the budget: Qwen3 stops at a user thresh­old [198], gpt-oss has three rea­son­ing levels [28], and Kimi k1.5 adds a group-rel­a­tive length reward [648]:

Lengths of 2,000, 4,000, 6,000 and 10,000 tokens give $+0.5$, $+0.25$, $0$ and $-0.5$ (com­puted, illus­tra­tive), and a wrong answer never gains. Think­ing can be too long or too short (Table 9.13).

Does sequen­tial com­pute need a checker? Setlur et al. prove that ver­i­fier-free meth­ods (fine-tuning on expert traces, as in s1) fall behind a simple ver­i­fier-based one by a gap grow­ing as $\sqrt{H}$ in the token budget $H$ when data grows with the budget [698]. Imi­ta­tion copies waste­ful traces too. A ver­i­fier keeps the cheap cor­rect ones. Together with the dis­til­la­tion results of Sec­tion 9.4, the rule is: dis­till to start, verify to con­tinue.

What RL adds: the pass@k argu­ment

Yue et al. com­pared base models with their RLVR ver­sions at $k$ up to 1,024 [484]. RL wins at small $k$, but at tens or hun­dreds of sam­ples the base passes it. For Qwen2.5⁠–⁠7B on AIME 2024 at $k=\mathrm{1,024}$, 13.3% of prob­lems were solved only by the base and 0.0% only by the RL model (reported). Count with the unbi­ased esti­ma­tor of Sec­tion 10.3, $1-\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ for $n$ sam­ples with $c$ cor­rect [699]: at $n=64$, $c=5$, $k=16$ it gives 0.775 where the naive $1-(1-c/n{)}^k$ gives 0.728 (com­puted).

Why would sharp­en­ing lose at large $k$? With $p_i$ the suc­cess prob­a­bil­ity on prob­lem $i$,

the share of prob­lems the model can solve at all. RL raises $p_i$ where it is pos­i­tive, lift­ing pass@1, and can push a few $p_i$ to zero, shrink­ing that sup­port. Take 30 illus­tra­tive prob­lems: under RL, 10 go from 0.30 to 0.80, 9 from 0.08 to 0.25, 4 from 0.01 to 0, and 7 stay unsolv­able. pass@1 is 0.125 for the base and 0.342 for RL, but the base leads from $k=32$ and ends at 0.767 against 0.633 (com­puted). In Figure 49, pick "Sharp­en­ing with­out for­get­ting" and the curves meet instead of cross­ing.

Table 9.14 shows the evi­dence splits by setup. As of Novem­ber 2026, RLVR at open-source scale mostly sharp­ens, and expands cov­er­age when trained long on prob­lems at the edge of com­pe­tence with entropy and KL man­aged (Sec­tion 9.6). A two-stage view, exploit then explore, rec­on­ciles the camps [700]. Dis­til­la­tion remains the reli­able way to add pat­terns.

9.8 RL as a systems problem: generation, the long tail and asynchrony

Why does an RL step on a large model use a few per­cent of the hard­ware's peak FLOP/s, and how do modern sys­tems run about twice as fast? A step sam­ples prompts, gen­er­ates answers in an infer­ence engine such as vLLM, scores them, com­putes old-policy and ref­er­ence log-prob­a­bil­i­ties, trains, and pushes new weights to the engine. Gen­er­a­tion is decod­ing, one token at a time, each read­ing every weight from memory, so it is bound by memory band­width (why roll­outs are decode-bound is in Sec­tion 13.2). It is the largest and least effi­cient part, and a syn­chro­nous step waits for its long­est answer.

Where the time goes

DeepSpeed-Chat put gen­er­a­tion at about 20% of the com­pu­ta­tion in 2023, with 256-token answers [705]. With long chains of thought, OpenRLHF's authors put infer­ence at over 90% of the run­time [706], and HybridFlow mea­sured NeMo-Aligner's gen­er­a­tion at up to 81.2% of an iter­a­tion [707]. Table 9.15 shows one break­down. For an illus­tra­tive 32B model and 32.8M response tokens, gen­er­a­tion at $2N$ per token is $2.1\times {10}^{18}$ FLOPs and the update at $6N$ is $6.3\times {10}^{18}$ (com­puted): three times the FLOPs, less of the clock. PipelineRL writes learn­ing speed as

reward per sample $S$ times sam­ples per second [686]. On-policy train­ing max­i­mizes the first factor, asyn­chrony the second.

The long tail, and why more GPUs do not fix it

Rea­son­ing lengths are heavy-tailed, and a syn­chro­nous step must wait for the long­est sequence [709]. As sequences finish, each GPU's decode batch drains, and StreamRL calls the idle time skew­ness bub­bles [710]. More GPUs leave each with fewer sequences, a memory-bound regime "where addi­tional devices fail to improve through­put" [709]. In Figure 50, watch the active sequences sag long before the last answer ends, then com­pare the three time­lines in step 4.

Memory adds pres­sure. Model states (Sec­tion 8.5) share GPUs with the KV cache, 128 KiB per token for Llama 3 8B in BF16 [6], so 128 sequences at 8,000 tokens need 134 GB, more than one 80 GB GPU (com­puted).

Two axes: place­ment and timing

Colo­cated sys­tems alter­nate the same GPUs between gen­er­at­ing and train­ing, dis­ag­gre­gated ones ded­i­cate GPUs to each. Syn­chro­nous sys­tems gen­er­ate with the cur­rent weights, asyn­chro­nous ones keep gen­er­at­ing while train­ing updates. Colo­ca­tion needs cheap reshard­ing, since train­ing wants wide model par­al­lel­ism and gen­er­a­tion narrow. HybridFlow gath­ers only within small groups where each GPU's train­ing shard is already part of its gen­er­a­tion shard [707, 711] (Table 9.17, with $a=p_{\text{tp}}{p}_{\text{pp}}$ the train­ing model-par­al­lel group and $a_g$ the gen­er­a­tion one, all-gather costs in Sec­tion 11.1). Its 1.53 to 20.57 times speedups forced equal response lengths, hiding the tail [707]. Kimi K2 stays colo­cated at 1T param­e­ters [72].

Dis­ag­gre­ga­tion gives memory-bound gen­er­a­tion and com­pute-bound train­ing their own GPUs and par­al­lel­ism [710], and asyn­chrony keeps both busy. AReaL admits a new gen­er­a­tion request only if

with $N_r$ the tra­jec­to­ries gen­er­ated so far, $B$ the train­ing batch in tra­jec­to­ries (not tokens, as else­where in this guide), $i$ the cur­rent policy ver­sion and ${\eta }_{\text{stale}}$ the stal­e­ness bound [709]. At $B=512$ and $i=0$ it admits 512, 1,024, 2,560 and 4,608 tra­jec­to­ries for ${\eta }_{\text{stale}}=0,1,4,8$ (com­puted). With the PPO clip cen­tered on a recent policy rather than the stale sam­pler, stal­e­ness up to 8 barely hurts, while naive PPO degrades with slight stal­e­ness [709]. Table 9.18 col­lects reported gains.

Why does asyn­chrony win (derived)? If gen­er­a­tion scaled per­fectly with GPUs, both designs would tie, since both keep every GPU busy. It does not: the end-of-batch bar­rier and the drain­ing batch waste the gen­er­a­tor, and stream­ing refills the batch while long answers finish. So par­tial roll­outs recover most of the gain with­out split­ting GPUs. Kimi con­tin­ues unfin­ished tra­jec­to­ries in the next iter­a­tion [648], and BubbleSpec (a 2026 pre­print) drafts future roll­outs in the tail's idle time [714].

To do it your­self (as of Novem­ber 2026), mea­sure the phase break­down and decode occu­pancy first. Colo­cate syn­chronously for short-to-medium rea­son­ing and go dis­ag­gre­gated and asyn­chro­nous for slow agent envi­ron­ments or very large models, as GLM-4.5 does by task, with FP8 roll­outs [295]. Keep stal­e­ness small: AReaL's 8, Magis­tral's ratio of 2 [668], OpenAI Five's 0 to 1 [715]. Log effec­tive sample size, train-infer­ence KL and sync time (Sec­tion 12.5). Frame­works include TRL [716], ROLL [717] and SkyRL [718]. Sizing sand­box fleets for RL is Sec­tion 14.7.

9.9 Agents and reward hacking

What changes when the policy acts over many turns in an envi­ron­ment, and what does an opti­mizer strong enough to write code do with a reward writ­ten by people? An agent's tra­jec­tory inter­leaves its own tokens, which are actions, with the envi­ron­ment's: tool out­puts, files, test logs. Rewards arrive at the end. And the reward is a pro­gram some­one wrote, with gaps that a strong opti­mizer will find if they are easier than the task.

From scaf­folds to out­come-reward RL

WebGPT trained GPT-3 in a text browser, and its answers beat human demon­stra­tors' 56% of the time [719]. ReAct's thought, action and obser­va­tion loop is still the default scaf­fold [720], and Tool­former kept a self-sam­pled API call only if it low­ered the loss on later tokens [721]. ReTool's RL rewards only the out­come and lets the policy learn when to run code mid-rea­son­ing [722], and SWE-Gym sup­plied 2,438 exe­cutable Python repos­i­tory tasks for train­ing SWE agents [723]. In 2025 out­come-reward RL reached soft­ware engi­neer­ing. SWE-RL rewards tex­tual sim­i­lar­ity to the true patch with­out run­ning code and solved 41.0% of SWE-bench Ver­i­fied at 70B [724]: it scales with­out sand­boxes but rewards close­ness, not cor­rect­ness. DeepSWE trained Qwen3⁠–⁠32B with RL alone on exe­cutable R2E-Gym tasks [725] to 42.2% pass@1 [9]. DeepSWE's setup (64 H100s, six days, 512 con­tain­ers per iter­a­tion) is in Sec­tion 14.7. Kimi K2 runs over 10,000 con­cur­rent sand­boxes [72].

Reward hack­ing

Spec­i­fi­ca­tion gaming is behav­ior that sat­is­fies the lit­eral objec­tive with­out the intended out­come, and it comes from a mis­spec­i­fied task, not a flawed RL algo­rithm [730]. More capa­ble agents exploit it more, some­times in abrupt phase tran­si­tions [731]. Over-opti­miza­tion of learned rewards is in Sec­tion 8.6. METR found an ear­lier o3 hack­ing in 39 of 128 RE-Bench runs, where the scorer was vis­i­ble, against 8 of 1,087 HCAST runs, about 41 times as often by these totals (com­puted, METR's post says more than 43 times), and telling it not to cheat barely helped [732]. Hack­ing can gen­er­al­ize: models that learned hacks in pro­duc­tion coding envi­ron­ments attempted sab­o­tage 12% of the time in MacDiarmid et al.'s main set­ting [610]. The cat­a­logue of doc­u­mented reward hacks is in Sec­tion 14.6.

Yet chains of thought often omit what drove a behav­ior (Table 9.19). A hypoth­e­sis rec­on­ciles 95% with under 2%: models may need to think out loud on some tasks with­out a robust propen­sity to on others [734], so mon­i­tor­ing works best where mis­be­hav­ior requires think­ing, as in Baker's long tasks. The gpt-oss devel­op­ers kept opti­miza­tion pres­sure off the chain of thought [28], but a 2026 pre­print finds mis­spec­i­fied rewards alone degrade mon­i­tora­bil­ity [735].

Mit­i­ga­tions sort by where they act. In the envi­ron­ment: graders the policy cannot touch, hidden held-out tests and impos­si­ble-task probes, as Anthropic used for Claude 4 [740]. In train­ing: mon­i­tors for alerts and patches, not reward [733], and inoc­u­la­tion prompt­ing, which frames hack­ing as accept­able during RL and cut final mis­align­ment by 75⁠–⁠90% [610] (a 2026 pre­print found the fram­ing fails in mid­train­ing doc­u­ments [741]). Telling the policy not to cheat and chat-style safety RLHF work poorly on agen­tic tasks [610]. Delib­er­a­tive align­ment hides the chain of thought from its RL judge [632], and Rule Based Rewards grade out­puts against writ­ten rules [631]. The common thread: reward out­puts and actions, and mon­i­tor the chain of thought instead.

A group of 8 answers has rewards (1, 1, 1, 1, 1, 1, 1, 0). What advan­tage does the one wrong answer get under GRPO, Dr. GRPO and RLOO, and why is GRPO's so large?

The mean is 0.875 and the pop­u­la­tion stan­dard devi­a­tion $\sqrt{0.875\times 0.125}=0.331$. GRPO gives $(0-0.875)/0.331=-2.65$, Dr. GRPO $-0.875$, and RLOO $0-1=-1.0$, since the other seven scored 1 (com­puted). Divid­ing by a small stan­dard devi­a­tion gives a rare fail­ure on an easy prompt a large weight. Over a group GRPO hands out $2G\sqrt{p(1-p)}$ of abso­lute advan­tage against $2Gp(1-p)$ for Dr. GRPO, a factor $1/\sqrt{p(1-p)}=3.0$ at $p=7/8$: the dif­fi­culty bias of Sec­tion 9.3.

GRPO adds $k_3=\delta -1-\log \delta$, with $\delta ={\pi }_{\text{ref}}/{\pi }_{\theta }$, to its loss. Is that a reverse-KL penalty?

Its value esti­mates the reverse KL, $\mathrm{K}\mathrm{L}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$, with­out bias. Its gra­di­ent does not. $\nabla k_3=(1-\delta )\nabla \log {\pi }_{\theta }$, and under ${\pi }_{\theta }$ the first term aver­ages to zero while the second aver­ages to $-{\sum }_y{\pi }_{\text{ref}}(y)\nabla \log {\pi }_{\theta }(y)=\nabla \mathrm{K}\mathrm{L}({\pi }_{\text{ref}}\Vert {\pi }_{\theta })$, the for­ward-KL gra­di­ent. For the reverse-KL gra­di­ent use $k_2=\frac{1}{2}(\log \delta {)}^2$ as the loss, or impor­tance-weight $k_3$ as DeepSeek-V3.2 does (Sec­tion 9.6).

A base model scores 12.5% pass@1 and its RL-tuned ver­sion 34%. At $k=\mathrm{1,024}$ the base scores 77% and the RL model 63%. Explain both num­bers with­out saying RL is bad.

pass@1 aver­ages per-prob­lem suc­cess prob­a­bil­i­ties, and RL raised them on prob­lems the base could already solve. At large $k$, pass@k tends to the share of prob­lems with any chance of suc­cess, and RL pushed a few of those prob­a­bil­i­ties to zero. It sharp­ened the dis­tri­bu­tion and shrank its sup­port. Caveat: on short-answer bench­marks some large-$k$ base hits are lucky guesses, and CoT-pass@k, which also checks the rea­son­ing, nar­rows or reverses the gap.

Why does adding GPUs to a syn­chro­nous RL job often barely speed up gen­er­a­tion?

The same batch over more GPUs leaves each with fewer sequences. Decod­ing at small batch is bound by memory band­width, since every step reads all the weights how­ever many sequences share them, so through­put per GPU falls. And the step cannot end before the long­est answer does: in the Llama 3 8B exam­ple that answer needs 120 s even alone. Asyn­chro­nous pipe­lines or par­tial roll­outs remove the bar­rier and keep decode batches full.

A chain-of-thought mon­i­tor flags reward hack­ing with 95% recall. Why not add its score to the reward?

Baker et al. found it helps at low opti­miza­tion, but with more opti­miza­tion the agent learned obfus­cated reward hack­ing and the mon­i­tor's recall fell to near zero. You lose the detec­tor and keep the hack­ing. Use the mon­i­tor to find and patch envi­ron­ment bugs, keep strong pres­sure off the chain of thought, and accept the mon­i­tora­bil­ity tax.

10. Evaluation: measuring what a model can do

A bench­mark score is a read­ing from an instru­ment with inspectable parts. The ques­tions, sam­pling pro­ce­dure, har­ness, grader and uncer­tainty deter­mine what the number means. An eval­u­a­tion is useful when it can resolve the deci­sion the team needs to make.

What does a number like "ARC-Chal­lenge 25-shot: 78.6" actu­ally tell you? Less than it seems. Meta's model card gives Llama 3 8B 78.6% on ARC-Chal­lenge, a set of grade-school sci­ence ques­tions. The Hug­ging Face Open LLM Leader­board gives it 60.2%. Both are 25-shot. The card scores the letter the model picks, the leader­board each answer string as a con­tin­u­a­tion [497]. Same weights, same bench­mark name, 18.4 points apart (com­puted).

The code mat­ters as much as the prompt. In 2023 LLaMA-65B scored 0.636 on MMLU with the bench­mark's orig­i­nal code, 0.637 with Stan­ford's HELM and 0.488 with the EleutherAI har­ness then behind the leader­board, a 14.8-point gap between the orig­i­nal code and the har­ness (com­puted). As the authors of that com­par­i­son put it, "eval­u­a­tions are strongly tied to their imple­men­ta­tions" [742].

So treat a score as a mea­sure­ment made with an instru­ment: a sample of ques­tions, an answer key, a prompt and for­mu­la­tion, the model and its decod­ing set­tings, a scor­ing rule or judge, and the aggre­ga­tion. Agents add a scaf­fold (the code that turns a model into an agent), an envi­ron­ment, a grader and an attempt budget. Each stage adds noise, which shrinks with more ques­tions, sam­ples or seeds, or bias, which does not. Figure 51 shows the largest doc­u­mented effect at each stage. Com­pare them with the dashed line at 2 points, a typ­i­cal gap between fron­tier models (illus­tra­tive).

Eval­u­a­tions make the deci­sions in this guide: data abla­tions (Sec­tion 3.9), mid­train­ing mixes (Chap­ter 7), post-train­ing recipes (Chap­ter 8 and Chap­ter 9), check­points and safe­guards (Sec­tion 10.8). Each needs the gap, the noise and the bias. Llama 3 pub­lishes 95% inter­vals and calls them a lower bound on the true vari­a­tion [6], and DeepSeek-R1 marks which stage-to-stage changes are sta­tis­ti­cally sig­nif­i­cant [3].

Sec­tions 10.1 to 10.4 map the bench­marks and open the instru­ment. Sec­tions 10.5 to 10.8 cover con­tam­i­na­tion, judges, agents and dan­ger­ous capa­bil­i­ties, and Sec­tion 10.9 builds the inter­nal suite. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

10.1 The benchmark landscape and how fast it saturates

Which num­bers do labs report, what does each one mea­sure, and why does every bench­mark stop being useful within a few years? Bench­marks come in fam­i­lies, each built just beyond the models of its day. You need the map, the sizes (they set error bars), the chance levels (they set the floor) and the life cycle.

Loss first

Loss on held-out text needs no answer key. Bits per byte, which com­pares losses across tok­eniz­ers, is defined in Sec­tion 3.8. Fit is not capa­bil­ity: DeepSeek-V3's base model has 0.548 bits per byte on the Pile test set against 0.542 for Llama 3.1 405B base, yet leads it on 5-shot MMLU, 87.1 to 84.4 (reported) [2]. Aver­age bench­mark score does rise almost lin­early with com­pres­sion across 31 models [743], and Sec­tion 6.7 pre­dicts down­stream scores from loss.

The base-model suite and its suc­ces­sors

Each bench­mark in Table 10.1 launched far below humans and closed the gap within a few years. HellaSwag's end­ings were triv­ial for people (over 95%) while the best model scored under 48% [744]. Test sets differ by a factor of about 470, from AIME 2024's 30 prob­lems to MMLU's 14,042 (the released test split, while the paper says 14,079), and Sec­tion 10.4 turns them into error bars.

Gen­er­a­tive math and code expose the answer checker. GSM8K's word prob­lems are frag­ile: adding one irrel­e­vant clause causes drops of up to 65% [761]. Weak unit tests inflate code scores, and HumanEval+ adds 80 times more tests and lowers pass@k by up to 19.3⁠–⁠28.9% [762].

Fron­tier sets buy head­room with expert dif­fi­culty. On GPQA, experts reach 65% and skilled non-experts with web access 34% [753]. DeepSeek-R1 reports 71.5% on its Dia­mond subset against 75.7% for OpenAI's o1⁠–⁠1217, but the o1 number comes from OpenAI's own reports, so the com­par­i­son crosses two pipe­lines [3]. HLE's authors warn that small changes near zero accu­racy are not strong evi­dence of progress [757], and once o3 reached 87.5% on ARC-AGI-1 with 172 times more com­pute, ARC Prize required cost to be reported [759].

Sat­u­ra­tion

Bench­marks die young. Ott et al. mapped 3,765 bench­marks and found that many trended quickly toward sat­u­ra­tion [763]. Akhtar et al. define sat­u­ra­tion as the loss of reli­able dis­crim­i­na­tion among the top models and score it with an index [754]:

Here $a_{(1)}$ and $a_{(5)}$ are the best and fifth-best scores as frac­tions, and $n$ ques­tions enter as $\sqrt{n}$ so that huge test sets do not look too dis­crim­i­nat­ing. An index near 1 means the top five are indis­tin­guish­able. On MATH-500 (top five from 99.2% to 98.2%, 500 prob­lems) the index matches the paper (com­puted):

Nearly half of their 60 bench­marks are sat­u­rated, from 42.9% of those under two years old to 54.5% of those over five (a modest trend that the authors note is not sta­tis­ti­cally sig­nif­i­cant), and pri­vate test sets did not pro­tect them [754].

The index is a con­ven­tion, not a test. On the MMLU row of DeepSeek-R1's com­par­i­son table, whose top five run from 91.8 to 87.2 on 14,042 ques­tions, it gives 0.26. With raw $n$ in place of $\sqrt{n}$ the same spread is 12.6 stan­dard errors and the index is about ${10}^{-69}$ (com­puted) [3]. Figure 52 fol­lows 16 of these bench­marks (GSM8K and ARC lack a release-time score in the sources read). In each panel, count the years from the first dot (the best score at release) to the dashed human line or the top. BIG-Bench Hard and AIME 2024 have no release-time point, and FrontierMath's caret is an upper bound. Hollow dots mark another subset or set­ting and stay off the run­ning-best line when the item set dif­fers.

Sat­u­rated bench­marks get replaced by harder suc­ces­sors in the same format (MMLU-Pro, ARC-AGI-2, BIG-Bench Extra Hard, where the best rea­son­ing model scores 44.8% [764]), expert adver­sar­ial sets (GPQA, FrontierMath, HLE), refreshed ques­tions (LiveBench updates monthly [765], and MathArena found strong signs of con­tam­i­na­tion in AIME 2024 [766]), and pref­er­ence and agent eval­u­a­tions (Sec­tions 10.6 and 10.7).

Three mis­takes recur: com­par­ing per-token loss across tok­eniz­ers, treat­ing MATH and MATH-500 (or GPQA and its Dia­mond subset) as one bench­mark, and read­ing a jump in exact-match accu­racy as a new abil­ity, since dis­con­tin­u­ous met­rics can pro­duce appar­ent emer­gence [477] (Sec­tion 6.7).

10.2 How a question becomes a number

Given a model and a bench­mark, which choices besides the model move the score, and by how much? A score is a fact about a model, a prompt, a scor­ing rule, an answer key and the code that joins them. Change any one and it moves, some­times by more than a model gen­er­a­tion.

Two for­mu­la­tions of a mul­ti­ple-choice ques­tion

The cloze for­mu­la­tion (CF) scores each answer string as a con­tin­u­a­tion of the ques­tion and picks the most prob­a­ble. The mul­ti­ple-choice for­mu­la­tion (MCF) shows let­tered options and scores only the letter pre­dicted next [497]. Raw log-prob­a­bil­ity penal­izes long answers, so cloze needs a nor­mal­iza­tion. For option $o_j$ and ques­tion $x$:

Llama models and the EleutherAI har­ness use the per-char­ac­ter form (the har­ness's acc_norm). The point­wise mutual infor­ma­tion (PMI) form divides out how likely the answer is after the bare prompt "Answer:". A per-token form also exists [497].

The right for­mu­la­tion depends on the model. MCF needs a learned skill, map­ping an answer to its letter, and OLMo-7B picked it up only after about 400B train­ing tokens, so cloze is the better early signal [497]. For strong models let­ters win: Llama 3 70B scores 93.7% on ARC-Chal­lenge with let­ters and 69.0% with cloze, error rates of 6.3% and 31% [497]. OLMES, the OLMo team's stan­dard, fixes the prompt, the 5-shot exam­ples and the nor­mal­iza­tion per task, runs both for­mu­la­tions and reports the better [497].

Prompt format

Mean­ing-pre­serv­ing changes of sep­a­ra­tors and spac­ing move accu­racy by up to 76 points for LLaMA-2 13B and about 10 points on aver­age over more than 50 tasks, and they flip order­ings: LLaMA-2 13B and LLaMA-2 70B reversed by at least 2 points with prob­a­bil­ity 0.141 [769]. Models also favor let­ters: on letter-bal­anced MMLU items llama-30B picks A 34.6% of the time and D 15.8% [770]. Design helps: across 24 prompts MMLU's range is usu­ally 4⁠–⁠5 points and MMLU-Pro's, with ten options, about 2 [125].

The answer key is a vari­able

MMLU-Redux re-anno­tated 100 ques­tions from each of 57 sub­jects and esti­mates that 6.49% of MMLU's ques­tions con­tain errors, includ­ing 57% of the Virol­ogy ques­tions it checked [771]. Weight­ing each sub­ject's error rate by its size in the 14,042-ques­tion test split repro­duces the 6.49% (com­puted). Llama 3.1 405B Instruct ranked 16th on all Virol­ogy ques­tions and first on the cor­rect ones [771]. Errors also cap the score. If a per­fect model matched the key on every sound ques­tion, a quar­ter of the unclear ones, half of those with sev­eral cor­rect options and none of the rest, it would score

(com­puted from MMLU-Redux's cat­e­gory shares, the match­ing shares are assump­tions). A score far above 95% sug­gests a mem­o­rized key.

The har­ness, the tem­plate and the decoder

Chat models need their own chat tem­plate [772]. Gen­er­a­tion caps fail silently: the GSM1k authors raised the har­ness's cap from 256 to 1,000 tokens because the default cut models off [773]. "Deter­min­is­tic" decod­ing varied accu­racy by up to 15% across runs for five models [774], and a dis­tilled 7B rea­son­ing model by up to 9% with GPU count, type and batch size [402]. Why iden­ti­cal prompts at tem­per­a­ture 0 give dif­fer­ent out­puts (80 dis­tinct out­puts from 1,000 com­ple­tions in Think­ing Machines' test) is in Sec­tion 12.7.

Every metric needs a triv­ial con­trol: Anthropic's BBQ bias eval­u­a­tion once showed a bias score of zero because the models were not answer­ing at all [772]. Table 10.2 col­lects this sec­tion's spreads, and Figure 51 sets them beside the other stages.

Prac­ti­cal defaults: track cloze or bits per byte early in pre­train­ing and let­ters for strong models. Report a format spread when you claim a small gap, use MMLU-Redux where a sub­ject's error rate is high, and pin the har­ness ver­sion.

10.3 Scoring many samples: pass@k, majority vote and pass^k

When a model can answer many times, how do you score it, and why are the obvi­ous for­mu­las wrong? Does at least one of $k$ attempts suc­ceed (pass@k, cov­er­age)? What is one sample's suc­cess rate (pass@1)? Do all $k$ suc­ceed (pass^k, reli­a­bil­ity)? Major­ity vote asks whether the most common final answer is right [691]. Here a bench­mark has $n$ ques­tions with $K$ sam­ples each (Chen et al. write $n$ for $K$), $c_i$ of ques­tion $i$'s sam­ples are cor­rect, and $a_i$ is its true per-sample suc­cess prob­a­bil­ity.

The unbi­ased pass@k esti­ma­tor

For one prob­lem with suc­cess prob­a­bil­ity $a$, true pass@k is $1-(1-a{)}^k$, the chance that $k$ inde­pen­dent sam­ples do not all fail. Chen et al. draw $K$ sam­ples (they use 200 with $k\le 100$), count the $c$ that pass the unit tests, and esti­mate pass@k as $1-\left(\genfrac{}{}{0ex}{}{K-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{K}{k}\right)$, which is unbi­ased [699]. The ratio is the chance that $k$ sam­ples drawn with­out replace­ment all fail. With $c$ drawn from Bino­mial$(K,a)$,

by the iden­tity $\left(\genfrac{}{}{0ex}{}{K-j}{k}\right)\left(\genfrac{}{}{0ex}{}{K}{j}\right)=\left(\genfrac{}{}{0ex}{}{K-k}{j}\right)\left(\genfrac{}{}{0ex}{}{K}{k}\right)$ and the bino­mial expan­sion of $(a+1-a{)}^{K-k}$. Aver­ag­ing over ques­tions, with a term equal to 1 when­ever $K-c_i<k$:

The prod­uct form on the right, from Chen et al.'s code, avoids huge bino­mial coef­fi­cients. The tempt­ing plug-in $1-(1-c/K{)}^k$ is biased low: $1-(1-x{)}^k$ is con­cave, so by Jensen's inequal­ity its aver­age over noisy esti­mates sits below its value at $a$, even when $K>5k$ [699]. For a true pass@1 of 0.02, 200 sam­ples and $k=100$, the unbi­ased esti­ma­tor aver­ages 0.8674, the true $1-{0.98}^{100}$, and the plug-in 0.7962 (com­puted). Feed­ing a bench­mark's mean pass@1 into the for­mula errs the other way: con­cav­ity makes it over­state pass@k when­ever prob­lems differ in dif­fi­culty. With dif­fi­cul­ties from Beta(0.3, 0.74) (illus­tra­tive, mean 0.29), true pass@10 is 0.61 and the pooled for­mula gives 0.97 (com­puted). Com­pute per prob­lem, then aver­age.

pass@k is not iden­ti­fied for $k>K$, how­ever many tasks you have [778]. And the best tem­per­a­ture depends on $k$: 0.2 for pass@1 and 0.8 for pass@100 for a 679M Codex model [699], so a pass@100 number hides a decod­ing choice.

pass^k: reli­a­bil­ity

tau-bench's pass^k is the chance that all $k$ inde­pen­dent trials of a task suc­ceed, aver­aged over tasks [779]. The with­out-replace­ment argu­ment again gives an unbi­ased esti­ma­tor, and the plug-in $(c_i/K{)}^k$ is biased upward because $x^k$ is convex. At $a=0.6$, $K=8$ and $k=2$ the esti­ma­tor aver­ages 0.3600, exactly ${0.6}^2$, and the plug-in 0.3900 (com­puted).

The inequal­ity on the right, Jensen again, is what pass^k reveals. gpt-4o with func­tion call­ing scores pass^1 of 61.2% on tau-bench's retail tasks, and its pass^8 drops below 25% [779]. If every task had suc­cess 0.612, pass^8 would be ${0.612}^8=2.0\%$ (com­puted). Mixed dif­fi­culty keeps pass^k high. If 24% of tasks always suc­ceed and the rest suc­ceed with prob­a­bil­ity 0.4895, pass^1 is $0.24+0.76\times 0.4895=0.612$ and pass^8 is $0.24+0.76\times {0.4895}^8=0.2425$ (com­puted, illus­tra­tive mix­ture). METR saw the split near a model's limits: on 90-minute to 3-hour tasks a GPT-5 agent always suc­ceeded on about a third, always failed on a third and was mixed on the rest [780]. In Figure 53, lower $c$ in one-prob­lem mode and watch the dashed plug-in lines leave the solid unbi­ased ones as $k$ grows. In pool mode, com­pare the dotted pooled short­cut with the true aver­age, out to $k=\mathrm{10,000}$ for the curves of Sec­tion 6.8.

The metric can reverse a ver­dict. Base models beat their RL-trained ver­sions at large $k$ across bench­marks and model fam­i­lies [484], so pass@1 and pass@256 can rank a train­ing method in oppo­site ways (Sec­tion 9.7). Prac­ti­cal defaults: the unbi­ased per-prob­lem esti­ma­tor with $K$ sev­eral times the largest $k$, pass@1 for rea­son­ing as a mean over sam­ples at a stated tem­per­a­ture, never greedy and sam­pled pass@1 in one table, pass^k when reli­a­bil­ity mat­ters, and gen­er­ated code run in a sand­box (Chap­ter 14).

10.4 Error bars: when is a gap real?

Model A scores 71% and model B 69% on the same bench­mark. Is the 2-point gap real, and how many ques­tions would you need to tell? Each ques­tion is a coin weighted by the model's skill, and an aver­age of coins wob­bles like one over the square root of their number. Two models face the same coins, so com­par­ing them ques­tion by ques­tion can­cels most of the wobble.

One score

Miller treats the ques­tions as "drawn at random from a (hypo­thet­i­cal, infi­nite, unseen) super-pop­u­la­tion of ques­tions" [777]. With ques­tion scores $y_i$ and mean $\widehat{a}$ over $n$ ques­tions, a 0/1 score has vari­ance $a(1-a)$, the mean divides it by $n$, and 1.96 is the normal 97.5th per­centile:

At 70%, the stan­dard error (SE) on 1,000 ques­tions is 1.45 points and the 95% half-width 2.84, against 9.0 for 100 ques­tions, 2.5 for GSM8K's 1,319 and 0.76 for MMLU's 14,042 (com­puted). Llama 3 prints this inter­val and warns that "our CI values lower bound the actual vari­a­tion in the capa­bil­ity esti­mate" [6]. The boot­strap only earns its keep for non­lin­ear aggre­gates [777].

Small sets and clus­tered sets

That inter­val (the Wald inter­val) assumes large $n$. At $n=30$ and a true 90%, it covers the truth 80.9% of the time, while the Wilson inter­val covers it 97.4% (com­puted exactly). Bowyer et al. find that "only the Bayesian cred­i­ble inter­val and Wilson con­fi­dence inter­vals" reach nom­i­nal cov­er­age on small sets [781]. With $z=1.96$:

For 30 of 30 cor­rect, Wald col­lapses to the point 100% and Wilson gives [88.6%, 100%] (com­puted).

Ques­tions about one pas­sage rise and fall together. With clus­ters of size $g$ and intra-clus­ter cor­re­la­tion ${\rho }_{\text{ICC}}$, a clus­ter's total has $g$ vari­ances plus $g(g-1)$ covari­ances, which gives the design effect (DE) [777]:

Miller mea­sured clus­tered-to-naive SE ratios of 3.05 for DROP (9,622 ques­tions in 588 pas­sages) and 1.88 for MGSM (2,500 in 250 groups) [777]. Invert­ing DE, DROP's ques­tions are worth about 1,034 inde­pen­dent ones (${\rho }_{\text{ICC}}=0.54$) and MGSM's about 707 (com­puted).

More sam­ples per ques­tion

With $K$ sam­ples per ques­tion and true suc­cess rate $a_i$, the law of total vari­ance splits the noise [777]:

Only the coin-flip term shrinks with $K$. Under uni­form dif­fi­culty, $K=4$ halves the vari­ance and no $K$ cuts it by more than two thirds (the floor is $\mathrm{V}\mathrm{a}\mathrm{r}(a_i)=1/12$ against $1/4$ at $K=1$). Greedy decod­ing "has inad­ver­tently tripled the min­i­mum vari­ance in the score data from 1/12 to 1/4" in the same exam­ple [777], and mea­sures a dif­fer­ent model.

Pair the com­par­i­son

Both models answer the same ques­tions, so take the per-ques­tion dif­fer­ence ${\Delta }_i=y_{A,i}-y_{B,i}$:

with ${\sigma }_A^2=a_A(1-a_A)$ and $\rho$ the cor­re­la­tion of the two models' out­comes across ques­tions. Models agree on which ques­tions are hard, so pair­ing is a "free" reduc­tion in vari­ance [777]. For 0/1 scores this is McNemar's view, where only dis­agree­ments carry infor­ma­tion [782]:

with $f_{10}$ the share A gets right and B wrong. At 71% against 69% and $\rho =0.8$, $f_{10}=0.052$, $f_{01}=0.032$, and both forms give 0.0840. On 1,000 ques­tions the paired SE is 0.92 points and $z=2.18$. Unpaired, the SE is 2.05 and $z=0.98$ (com­puted). Same data, oppo­site ver­dicts.

Power and sample size

Power is the prob­a­bil­ity of reject­ing the null when the effect is real [782]. For a true gap $\delta$, $\widehat{\Delta }/\mathrm{S}\mathrm{E}$ has mean $\delta /\mathrm{S}\mathrm{E}$, which must clear $z_{1-\alpha /2}+z_{\text{power}}$ (1.96 + 0.84 for 80%). Solve for $n$ [777]:

Miller's own exam­ple needs 969 ques­tions, so he sug­gests new evals hold at least 1,000 [777].

At $\rho =0.8$ a true 2-point gap needs about 1,650 ques­tions, a fifth of the unpaired count, and a ±2-point inter­val (807 ques­tions) catches it only about half the time (com­puted). Only 46% of the state-of-the-art claims Card et al. sur­veyed would have pre­dicted ade­quate power [782].

In Figure 54, the "One score" tab draws Wald and Wilson inter­vals, and "Two models" plots ques­tions needed against $\rho$ on a log axis, with lines at real bench­mark sizes. Count how many fall below the curve for a 2-point gap.

Many com­par­isons and selec­tion

Com­pare two equal models on 20 bench­marks and at least one "sig­nif­i­cant" gap appears with prob­a­bil­ity $1-{0.95}^{20}=0.64$. The best of 27 equal vari­ants car­ries about 2 SEs of pure noise, the expected max­i­mum of 27 stan­dard nor­mals (com­puted). Report how many vari­ants you tried. Hand-curated bench­marks are not exchange­able draws, and Bowyer et al. prefer Bayesian inter­vals for small sets [781].

# keep per-question scores for every run
import numpy as np
from scipy.stats import norm

def paired_gap(y_a, y_b):
    """Per-question scores (0/1 or partial) of two models, same questions."""
    delta = np.asarray(y_a, float) - np.asarray(y_b, float)
    se = delta.std(ddof=1) / np.sqrt(len(delta))
    return delta.mean(), se, delta.mean() / se      # gap, paired SE, z

def questions_needed(var_delta, gap, alpha=0.05, power=0.80):
    z = norm.ppf(1 - alpha / 2) + norm.ppf(power)
    return int(np.ceil(z**2 * var_delta / gap**2))

questions_needed(0.0840, 0.02)   # 1649, the rho = 0.8 row of the table

Report $n$, the score and its SE, and for com­par­isons the paired SE and $\rho$. If the expected effect is below the min­i­mum detectable one, use a bigger bench­mark or a con­tin­u­ous metric. Seed noise comes on top (Sec­tion 10.9).

10.5 Contamination: has the model seen the test?

A model scores 85% on a bench­mark whose ques­tions have been on the public web for years. How much of that is skill, and how much is memory of the test? Web-scale data almost cer­tainly holds copies and para­phrases of them.

Keep four terms apart. Con­tam­i­na­tion means test items or close vari­ants in any train­ing data, syn­thetic data included [783]. Exploita­tion means it raised the score, which mem­o­riza­tion need not do [784]. Over­fit­ting to a bench­mark is a gap between the public test and a fresh one, what­ever the cause. Indi­rect con­tam­i­na­tion comes from bench­marks pasted into prod­ucts that feed train­ing [785].

Audits with the train­ing data

GPT-3 flagged 13-gram over­laps and com­pared clean sub­sets with full sets [36]. Llama 2 flagged tokens in shared n-grams longer than 10 tokens, find­ing only HellaSwag and MMLU-Human­i­ties boosted [229]. Llama 3 scores 8-gram over­lap and, fol­low­ing ConTAM (Singh et al.'s Con­tam­i­na­tion Thresh­old Anal­y­sis Method), picks each bench­mark's thresh­old to max­i­mize the esti­mated gain, the clean-subset score minus the full score [6, 786]. ConTAM warns that con­tam­i­na­tion "may have a much larger effect than reported in recent LLM releases" [786].

Para­phrases and closed models

String match­ing misses rephras­ing. A Llama-2⁠–⁠13B trained on a rephrased MMLU test set reached 85.9 "while being unde­tectable by n-gram over­lap", and the syn­thetic CodeAlpaca set con­tains 12.8% rephrased HumanEval sam­ples [504].

With­out the train­ing data you can only probe prob­a­bil­i­ties. Seen text has fewer very unlikely tokens, so Min-K% Prob aver­ages the log-prob­a­bil­i­ties of the least likely 20% of a text's tokens, the set $S$:

It reaches an aver­age AUC of 0.72 on WikiMIA [787] (Min-K%++ refines it [788]). Treat such scores as weak evi­dence: on Pythia models, mem­ber­ship-infer­ence attacks "barely out­per­form random guess­ing for most set­tings" [789]. A sharper test: a model that mem­o­rized a bench­mark file prefers its canon­i­cal exam­ple order, which reli­ably reveals 1,000-exam­ple sets seen 10 or more times and some­times sets seen only twice. It found little per­va­sive con­tam­i­na­tion in four public models [790].

Mea­sure the effect: fresh repli­cas

GSM1k is 1,205 new prob­lems matched to GSM8K, so sim­i­lar that anno­ta­tors picked it out of five only 21.83% of the time against 20% chance [773]. Yi-6B-Chat scores 8.0 points lower on it (43.7% to 35.7%) and Meta-Llama-3⁠–⁠8B-Instruct 6.2. Phi, Mis­tral and some Llama models seem to over­fit, while Gemini, GPT and Claude show "little to no signs of over­fit­ting" [773]. The sets differ, so the gap is unpaired:

For Yi-6B-Chat on 1,319 and 1,205 prob­lems this is 1.94 points, a 95% inter­val of 4.2 to 11.8. Of 63 models, 17 have inter­vals entirely above zero. Under the paper's alter­na­tive prompt 12 do, and only 5 clear zero under both (com­puted), so a single-prompt gap is weak evi­dence on its own. The gap rises with a model's log-like­li­hood of GSM8K (Spear­man 0.36, p = 0.03), which the authors read as par­tial mem­o­riza­tion, though out­liers lead them to say con­tam­i­na­tion is "likely not the full story" [773].

In Figure 55, red rows have inter­vals clear of zero. They clus­ter by family, while fron­tier models strad­dle the "no drop" line.

Pre­ven­tion, and what to report

Defenses stack. BIG-bench files carry a canary string for fil­ter­ing [791], and Jacovi et al. pro­pose encrypt­ing public test data [792]. Post-cutoff and live sets help for a while (Sec­tion 10.1). Tülu 3 flags a post-train­ing set whose instances over­lap more than 2% of an eval­u­a­tion's instances. It drops the whole set, or for devel­op­ment eval­u­a­tions only the match­ing instances when drop­ping the set would hurt the model [550]. One chan­nel escapes every train­ing audit: search-enabled agents can retrieve answers at test time, inflat­ing scores by up to 4% [793].

For every head­line bench­mark, report the over­lap method, its thresh­old and the clean-subset score. Keep a pri­vate replica, and decon­tam­i­nate super­vised and pref­er­ence data, syn­thetic data included.

10.6 Judges, arenas and human preference

For open-ended answers with no answer key, how do LLM judges and human-pref­er­ence arenas pro­duce a rank­ing, and what biases and noise enter? A judge, human or model, is an instru­ment with noise, bias and a cal­i­bra­tion curve. Mea­sure all three.

LLM judges and their biases

Judges exist because "our best auto­mated tools for deter­min­ing whether two sen­tences are seman­ti­cally equiv­a­lent are the very models we are seek­ing to eval­u­ate" [775]. On MT-Bench's 80 ques­tions, GPT-4 agrees with experts 85% of the time with­out ties, above the 81% human-human rate, and agree­ment rises from 70% to nearly 100% as the gap between models grows [794]. Judges are reli­able on big gaps and weak on the close calls where deci­sions live, and harder tests are less flat­ter­ing [795, 796].

Fixes: judge both orders and "only declare a win when an answer is pre­ferred in both orders" [794], give a ref­er­ence answer where one exists, use a judge from another family, and con­trol for length.

Cal­i­brate the judge with a few human labels

Pre­dic­tion-pow­ered infer­ence (PPI) removes a judge's bias with a small human sample [802]. With judge label $J$ and human label $G$, aver­age the judge over $n_J$ items and sub­tract its mean error on $n_G$ human-labelled ones:

Its expec­ta­tion is $𝔼[J]-(𝔼[J]-𝔼[G])=𝔼[G]$, what­ever the bias. In an illus­tra­tive sim­u­la­tion, the true win rate is 0.55 and the judge says "A" 85% of the time when humans prefer A and 30% oth­er­wise. The judge alone esti­mates 0.60 and its inter­val never covers the truth. PPI with 10,000 judged and 300 human pairs covers it about 95% of the time, about 10.6 points wide against 11.3 from humans alone (com­puted, 2,000 repli­cates).

Arenas and the Bradley-Terry model

In Chat­bot Arena a user prompts two anony­mous models and votes. By Jan­u­ary 2024 it had about 240K votes, and crowd votes matched experts 72⁠–⁠83% of the time [803]. The Bradley-Terry (BT) model gives each model a strength $\xi$, shown on the Elo scale $R$:

Arena fits it by reweighted logis­tic regres­sion with sand­wich stan­dard errors [803], replac­ing online Elo updates, which are volatile [804]. For $n$ votes between two models at win rate $w$, the delta method gives

so 1,000 votes at $w=0.5$ give a 95% half-width of 21.5 Elo (com­puted). The Arena tab of Figure 54 runs these num­bers.

Length and style

On AlpacaEval the base­line's own win rate "fluc­tu­ates from 22.9% to 64.3%" with the ver­bosity instruc­tion alone, and a length-con­trolled regres­sion raises the cor­re­la­tion with Arena "from 0.94 to 0.98" [603]. Arena's style con­trol finds length "the dom­i­nant style factor" [806]. Over-opti­miza­tion of learned rewards is in Sec­tion 8.6.

Selec­tion on a public leader­board

BT assumes each score is one honest esti­mate. "The Leader­board Illu­sion" reports 27 pri­vate vari­ants tested by Meta before Llama 4, and esti­mates that Google and OpenAI received 19.2% and 20.4% of Arena data, against 29.7% for 83 open-weight models together. That data mat­ters: rais­ing its train­ing share from 0 to 70% lifted ArenaHard win rate from 23.5% to 49.9%. Of 243 public models, 205 aver­aged 10 or fewer bat­tles between March 3 and April 23, 2025, which the authors call silent dep­re­ca­tion [805]. Votes can be manip­u­lated [811], and BT is mis­spec­i­fied for near-copies [812]. This guide did not review the orga­niz­ers' responses.

If each vari­ant's score car­ries inde­pen­dent noise with a stan­dard devi­a­tion of 10 Elo, pub­lish­ing the best of 5, 10 or 27 adds about 11.6, 15.4 or 20.0 Elo (com­puted). Singh et al.'s sim­u­la­tion, with a dif­fer­ent noise model, finds about 50 points for 20 vari­ants [805].

Human eval­u­a­tion design

Llama 3 used about 7,000 prompts on a 7-point pair­wise scale, and "Mod­el­ing teams did not have access to our human-eval­u­a­tion prompts" [6]. Three work­ers rating 100 items is under­pow­ered for small effects [782]. Detect­ing a 52% win rate against 50% with 80% power takes about 4,900 non-tied com­par­isons (com­puted).

In prac­tice, swap and agree, con­trol length, use a cross-family judge, val­i­date it on 200⁠–⁠500 of your own human labels and cor­rect with PPI. Rank only where inter­vals sep­a­rate.

10.7 Evaluating agents: environments, scaffolds, graders and time horizons

When the unit of eval­u­a­tion is a multi-step tra­jec­tory in an envi­ron­ment, what exactly is being mea­sured, how trust­wor­thy is the number, and how do you com­pare agents across very dif­fer­ent tasks? The thing mea­sured is a system: the model, the scaf­fold (the loop that turns model out­puts into tool calls), the envi­ron­ment, the grader, and the budget in steps, tokens, dol­lars and attempts [813]. Change any part and the score moves.

A grader inspects the final state, not the tran­script: hidden unit tests for SWE-bench, the final data­base for tau-bench, the final con­tainer for Ter­mi­nal-Bench [814, 779, 815]. Inspect frames an eval­u­a­tion as a dataset, a solver and a scorer, with model code in sand­boxes [816], the sub­ject of Chap­ter 14.

SWE-bench, the run­ning agen­tic exam­ple

SWE-bench has 2,294 tasks from GitHub issues in 12 Python repos­i­to­ries. A task is resolved only if the patch applies and all asso­ci­ated tests pass [814]. How SWE-bench grades and how Ver­i­fied was built is in Sec­tion 14.4. For mea­sure­ment, note three things. Screen­ing for Ver­i­fied removed 68.3% of anno­tated sam­ples, and on the 500 left GPT-4o rose from 16% to 33.2%, one seed per scaf­fold (reported) [16]. Django sup­plies 231 of the 500, so by the inverse Simp­son index the set behaves like 3.9 equal-sized repos­i­to­ries (com­puted from the released dataset) [817]. And 24% of the top-50 leader­board posi­tions are still wrong because tests miss edge cases [818]. DeepSeek-V3 reports 42.0% and DeepSeek-R1 49.2%, both through Agent­less, a fixed local­ize, repair and val­i­date pipe­line with no agent loop (reported) [2, 3, 819]. Table 10.7 lists other suites.

Scaf­folds, bud­gets and graders

David­son et al. price an enhance­ment as a com­pute-equiv­a­lent gain (CEG), the extra train­ing com­pute that would buy the same improve­ment [828]. At $\lambda$ points per ten­fold com­pute, $\Delta a$ points are worth $\text{CEG}={10}^{\Delta a/\lambda }$: at an illus­tra­tive 10 points per decade, +7 points is 5.0 times (com­puted). Most enhance­ments they sur­veyed were worth over 5 times and cost under 1% of train­ing [828]. Retry­ing raises accu­racy too, so plot accu­racy against dol­lars [813]. On MLE-bench the scaf­fold alone moves GPT-4o from 0.8% to 8.7% [824], though Ter­mi­nal-Bench finds model choice usu­ally mat­ters more [815]. Report the scaf­fold as part of the system.

A grader has out­come valid­ity if a pass truly means the task was done. Seven of ten audited agen­tic bench­marks had out­come-valid­ity flaws, and an agent that returns empty responses passes 38% of tau-bench's air­line tasks [818]. The cat­a­logue of doc­u­mented reward hacks is in Sec­tion 14.6.

METR's time hori­zon

How do you com­pare agents across very dif­fer­ent tasks? METR mea­sures dif­fi­culty in human time. The 50% time hori­zon $H_{50}$ is "the time humans typ­i­cally take to com­plete tasks that AI models can com­plete with 50% suc­cess rate" [486], a mea­sure of task dif­fi­culty, not of how long the AI works [780]. Task length $t$ is the geo­met­ric mean of suc­cess­ful human times, and per agent METR fits a logis­tic in log task length [486], writ­ten here with slope $\gamma$ per dou­bling (the paper's ${\beta }_{\text{agent}}$):

The second form, from solv­ing $P=q$, gives the hori­zon at reli­a­bil­ity $q$. At METR's typ­i­cal slope of "roughly 0.6" [829], $H_{80}=H_{50}/5.0$ (com­puted), match­ing the paper's 4 to 6 times [486]. METR reports no 99% hori­zon, which would need many more tasks [780].

Regress­ing ${\log }_2{H}_{50}$ on release date gives a dou­bling time $T_d$ that depends on the suite ver­sion and the years fitted. The paper found 207 days (95% inter­val 166 to 240) [486]. Under Time Hori­zon 1.1 the trend since 2023 dou­bles every 131 days [487]. METR cor­rected a reg­u­lar­iza­tion mis­take on March 3, 2026, after that post [780]. Our refit on METR's pub­lished data, fron­tier models up to 16 hours, gives 187.8, 128.7 and 104.7 days for all years, 2023 on and 2024 on (com­puted) [780]. Say "about three to seven months (as of Sep­tem­ber 2026)". Figure 56 has a fit-window toggle: watch the line steepen as early years drop, com­pare whiskers with the gaps between models, and note the band above 16 hours, where METR calls mea­sure­ments unre­li­able [780].

METR's own caveats matter. Errors are cor­re­lated across models, so the slope is better known than any single hori­zon [486]. "An 8-hour time hori­zon does not mean that AIs can do 8 hours of work that a (high-con­text) human pro­fes­sional can do as part of their day-to-day job" [780]. In a 2025 trial, expe­ri­enced devel­op­ers took 19% longer with AI tools while believ­ing they were 20% faster [830]. A ten­fold error in every hori­zon shifts arrival by only $T_d{\log }_{2}10$, 1.2 years at 128.7 days (com­puted).

Agent suites are small. The 95% half-width is 4.0 points at 70% on Ver­i­fied's 500 tasks and 10.0 points at 63% on Ter­mi­nal-Bench 2.0's 89 (com­puted) [16, 815]. Pair on tasks, and clus­ter by repos­i­tory.

10.8 Dangerous-capability evaluations and frontier safety frameworks

How do labs decide whether a model is too capa­ble to release with ordi­nary safe­guards, and what eval­u­a­tion evi­dence feeds that deci­sion? This sec­tion stays at policy and bench­mark level. Shevlane et al. sep­a­rate dan­ger­ous-capa­bil­ity eval­u­a­tions (can the model do harm) from align­ment eval­u­a­tions (would it), and note that inter­nal deploy­ments, such as coding assis­tants for AI researchers, may also need eval­u­a­tion before use [831].

Google DeepMind's pilot on Gemini 1.0 cov­ered per­sua­sion and decep­tion, cyber-secu­rity, self-pro­lif­er­a­tion and self-rea­son­ing, and found no strong dan­ger­ous capa­bil­i­ties but early warn­ing signs [832]. To see progress before full suc­cess, it charges bits of expert help per mile­stone [832]:

A mile­stone passed in 10% of unguided runs costs 4 bits, and an expert pick­ing the 16th of 16 sam­pled actions costs 8.1 (com­puted). The charges are code lengths of prob­a­bil­i­ties that sum to 1. METR's 2023 report found agents com­pleted only the eas­i­est of 12 auton­omy tasks, yet said this did not assure that a next gen­er­a­tion at about 100 times the effec­tive com­pute would lack the capa­bil­ity [833]. That is the logic behind safety buf­fers. Public bench­marks are prox­ies that do not spread the hazard: WMDP's 3,668 fil­tered ques­tions track a pri­vately held sen­si­tive set [834], and Cybench cal­i­brates 40 cap­ture-the-flag tasks by human first-solve times [835]. Table 10.8 com­pares three frame­works.

Why a lower bound? Fron­tier models can be prompted to under­per­form selec­tively on dan­ger­ous-capa­bil­ity eval­u­a­tions [843], and a few demon­stra­tions often fine-tune back capa­bil­i­ties hidden behind a pass­word [844]. The o1 system card rated the model medium risk before mit­i­ga­tion, called its eval­u­a­tions a lower bound and noted pre-deploy­ment test­ing by the US and UK AI Safety Insti­tutes [671]. Its cap­ture-the-flag sand­box case is in Sec­tion 14.1.

10.9 The internal eval suite: signal, noise and decisions

A train­ing team runs thou­sands of eval­u­a­tions to choose data mixes, hyper­pa­ram­e­ters and check­points. How do you build a suite whose small-scale results pre­dict the large model, and keep your own team from gaming it? Sec­tion 10.4 treated the ques­tions as random. In devel­op­ment the run is random too. A bench­mark helps a deci­sion only if the gap between options (signal) is large against seed and check­point noise.

Madaan et al. trained ten 7B models for 210B tokens that differ only in seed. Seed stan­dard devi­a­tions (SD) were 0.57 points on MMLU, 0.21 on HellaSwag, 0.80 on ARC-Chal­lenge and 1.11 on HumanEval [745], for one 7B setup with a fixed data order, not typ­i­cal noise for strong models. Com­par­ing two single-seed runs adds seed noise from both to the paired ques­tion error:

On MMLU that is 0.81 points from seeds alone (com­puted). DataDecide saw run-to-run SDs up to 2 points at 1B [258].

Heine­man et al. define signal-to-noise ratio (SNR) as the rel­a­tive spread across models trained at the same com­pute, over the rel­a­tive SD of one model's final check­points [259]:

SNR pre­dicts deci­sion accu­racy, the share of pairs whose small-scale rank­ing holds at large scale ($R=0.791$), while signal or noise alone do not. Prun­ing noisy sub­tasks and aver­ag­ing the last check­points each add over 2 points of deci­sion accu­racy, and scor­ing bits per byte on the gold answer lifts GSM8K's SNR from 1.2 to 7.0 [259]. Bits per byte, which com­pares losses across tok­eniz­ers, is defined in Sec­tion 3.8. Con­tin­u­ous met­rics and cloze for­mats (Sec­tion 10.2) also suit pre­train­ing abla­tions better than letter choices [745]. Rank­ing at 150M picked the 1B winner about 80% of the time, and no scal­ing-law method beat it [258]. Tiny 100-item sub­sets raised HellaSwag's seed SD about ten­fold, from 0.21 to over 2 points, and flipped up to 12% of pair­wise com­par­isons [745], so mon­i­tor with them and decide with full sets.

A lay­ered suite fol­lows: smooth in-loop bits-per-byte sig­nals, deci­sion bench­marks chosen by mea­sured SNR, capa­bil­ity bench­marks once above chance, a held-out set never used for deci­sions, and val­i­dated human and judge eval­u­a­tions (Sec­tion 10.6). OLMo 2 declares devel­op­ment and held-out tasks [7]. DeepSeek-R1's per-stage table shows what a head­line hides: from R1-Zero to R1, IF-Eval rose from 46.6 to 83.3 while GPQA Dia­mond fell from 75.8 to 71.5 [3], a 4.3-point drop, about one stan­dard error of the unpaired dif­fer­ence (4.4 points on 198 ques­tions, com­puted).

This is Good­hart's law: "When a mea­sure becomes a target, it ceases to be a good mea­sure" [575]. Its four vari­ants [846] map onto best-of-many report­ing (regres­sional), bench­mark-chas­ing (extremal), train­ing on the test (causal) and gaming a judge (adver­sar­ial). Reward-model overop­ti­miza­tion is the clean­est case (Sec­tion 8.6). Keep a deci­sion log (Sec­tion 15.1). Eval­u­a­tion gates, lin­eage and ver­sion­ing for long runs are in Sec­tion 12.9.

A paper reports pass@100 = 0.78 from the plug-in $1-(1-c/K{)}^{100}$ with $K=200$. Is the true pass@100 likely higher or lower?

Higher. $1-(1-x{)}^k$ is con­cave, so by Jensen's inequal­ity the plug-in is biased low: at true suc­cess 0.02 it aver­ages 0.796 against 0.867 [699]. Use the unbi­ased esti­ma­tor per prob­lem (Sec­tion 10.3).

Model A scores 71% and B 69% on 1,000 ques­tions, per-ques­tion cor­re­la­tion 0.8. Is the gap resolved?

Unpaired, SE 2.05 points and $z\approx 1.0$. Paired, SE $\sqrt{0.084/1000}=0.0092$ and $z\approx 2.2$, sig­nif­i­cant, but power for a true 2-point gap is only about 59%. 80% power needs about 1,650 ques­tions (com­puted) [777].

Llama 3's audit flags 52% of Nat­u­ral Ques­tions as con­tam­i­nated yet esti­mates almost no gain. How?

Con­tam­i­na­tion is not exploita­tion: over­lap­ping text can lack the answer, or seeing an item may not help [6, 784]. Flagged and clean sub­sets also differ in dif­fi­culty.

An agent's 50% hori­zon is 2 hours. Roughly what is its 80% hori­zon, and why does no one report a 99% hori­zon?

About 24 min­utes: at slope 0.6, 80% suc­cess needs $logit(0.8)/0.6=2.3$ dou­blings shorter tasks [829]. A 99% hori­zon would be about 200 times shorter at that slope, and 100 to 2,300 times across METR's slopes (com­puted), and would need many very short, reli­able tasks [780].

You must choose between two data mixes with 150M-param­e­ter models. What makes a bench­mark trust­wor­thy for that?

High SNR: recipe spread large against check­point noise. SNR tracks deci­sion accu­racy ($R=0.79$), and bits per byte raised GSM8K's SNR from 1.2 to 7.0 [259]. Rank­ing at 150M picked the 1B winner about 80% of the time [258].

11. Distributed training: from one GPU to a hundred thousand

Dis­trib­uted train­ing is an account­ing prob­lem for bytes: which device owns them, which link they cross, how often they move, and how much move­ment can over­lap with com­pute. Each par­al­lel axis trades memory, com­mu­ni­ca­tion and sched­ul­ing against the others.

Why can one GPU not train a fron­tier model, and in how many ways can the work be split? Three walls stand in the way. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

Time. Llama 3 405B took $3.8\times {10}^{25}$ FLOPs, at 400 TFLOP/s per GPU on 16,384 H100s [6]. One GPU would need $3.8\times {10}^{25}/(4\times {10}^{14})=9.5\times {10}^{10}$ sec­onds, about 3,000 years, and the 16,384 GPUs need about 67 days (com­puted). Memory. At 16 bytes per param­e­ter (BF16 weights and gra­di­ents, FP32 master weights and Adam moments) [22], 405.85B param­e­ters need 6.49 TB of train­ing state, more than 81 H100s of 80 GB can hold, before any acti­va­tion (com­puted) [4]. Batch. If each GPU held the whole model and one 8,192-token sequence, 16,384 GPUs would make a 134M-token batch, eight times the 16M Llama 3 used (com­puted). Past the crit­i­cal batch size a bigger batch stops help­ing [360] (Sec­tion 5.3). So the work is split along five axes, and big runs com­bine sev­eral of them (Table 11.1).

Llama 3 states the order­ing rule: "The inner­most par­al­lel­ism requires the high­est net­work band­width and lowest latency, and hence is usu­ally con­strained to within the same server" [6]. On an H100 server, NVLink car­ries 900 GB/s in total, about 450 GB/s each way per GPU, while each GPU's 400 Gb/s net­work card car­ries 50 GB/s, a factor of 9 (com­puted) [4, 137]. Chap­ter 2 describes the hard­ware.

After col­lec­tives (Sec­tion 11.1) and data par­al­lel­ism on nanoGPT and Llama 3 70B (Sec­tion 11.2) come the model splits, with DeepSeek-V3 for pipe­lines and experts, and Llama 3 405B's full layout (Sec­tion 11.7). The "hun­dred thou­sand" is the range of the meth­ods: the largest layout with pub­lished degrees here uses up to 16K GPUs of a 24K-GPU clus­ter [6].

11.1 Collectives: the vocabulary and the cost model

What com­mu­ni­ca­tion pat­terns does par­al­lel train­ing need, and how long does each take? Every axis talks through col­lec­tives, oper­a­tions that all ranks (pro­cesses, usu­ally one per GPU) in a group call together on same-shaped arrays [143, 847].

Float­ing-point addi­tion is not asso­cia­tive [848], and in a ring each chunk's sum starts at a dif­fer­ent rank (our read­ing of the sched­ule below), so chang­ing the rank count changes the last bits. Llama 3 reduces gra­di­ents in FP32 [6] (deter­min­ism is in Sec­tion 12.7).

The alpha-beta model and the ring

A mes­sage of $n$ bytes costs $t=\alpha +n/W$: a fixed latency $\alpha$ plus bytes over the per-direc­tion band­width $W$, "roughly equal to over­head + size / bw" in the nccl-tests doc­u­men­ta­tion [143, 849]. Now put $p$ ranks on a ring, each send­ing only to its right neigh­bor, and cut the array into $p$ chunks [850]. In the reduce-scat­ter phase, at step $k$ (from 0), rank $r$ sends chunk $(r-k)modp$ to rank $r+1$, which adds it in. After $p-1$ steps each rank owns one fully summed chunk. The all-gather phase repeats the rota­tion, but the receiver over­writes instead of adding. Each step moves $n/p$ bytes per rank, so

A sim­u­la­tion for $p$ from 2 to 16 gives exact sums (com­puted). In Figure 57, choose four GPUs and step: each chunk's sum grows from one con­tri­bu­tion to four, then the fin­ished chunks spread. The side chart shows bytes per GPU near­ing twice the mes­sage while steps grow lin­early.

Point-to-point algo­rithms cannot do much better: "the best of any all-reduce algo­rithm can do is to achieve roughly 2 times the time to send the msize-byte data" [851, 143]. The band­width factor $2(p-1)/p$ is 1.75 at 8 ranks and 1.998 at 1,024 (com­puted), so the ring's cost is nearly "inde­pen­dent of the number of GPUs in the system" [850]. The latency term, $2(p-1)\alpha$, is what grows.

Latency, trees and hier­ar­chy

Equat­ing the two terms gives the size below which a ring is latency-bound, $n^{*}=p\alpha W$: 2 MB at 8 ranks, 256 MB at 1,024 and 4 GB at 16,384 with $\alpha =5$ µs (illus­tra­tive) and 50 GB/s (com­puted). NCCL 2.4 added trees because "latency scales lin­early with the number of GPUs, pre­vent­ing scal­ing above hun­dreds of GPUs" [852]. A double binary tree moves about the bytes of a ring in log­a­rith­mi­cally many steps, cut latency up to 180 times at 24K GPUs on Summit, and NCCL falls back to rings when they give more band­width [852, 853].

Libraries also exploit the factor of 9 between NVLink and the net­work. NCCL's hier­ar­chi­cal ring "per­forms a reduce-scat­ter oper­a­tion inside the node, then mul­ti­ple all-reduce oper­a­tions between nodes, then an all-gather oper­a­tion inside the node again" [852], and the scal­ing book counts about 400 GB/s of net­work egress per 8-GPU node [138]. For $q$ nodes of $G$ GPUs:

So layout sets band­width. A group with one GPU per node, like the data-par­al­lel group when tensor par­al­lel­ism fills each node, pushes each GPU's whole mes­sage through its own 50 GB/s card. A group of whole nodes sends $1/G$ of it per card.

Read­ing nccl-tests

nccl-tests report algo­rithm band­width, algbw $=S/t$, and bus band­width, which rescales it by the lower bound so it com­pares with the link rate [143]:

An all-reduce at 200 GB/s of algbw on 8 GPUs shows 350 GB/s of busbw (com­puted). Pre­dict times with algbw, judge hard­ware with busbw. Inside an H100 node "it is dif­fi­cult in prac­tice to exceed 370 GB/s" [138]. NCCL's tuner picks low-latency pro­to­cols (about 1 to 2 µs per hop) for small mes­sages and Simple (about 6 µs, near peak band­width) for large ones [854].

In Figure 58, start from the 70B preset and slide the GPU count: above 64 GPUs the flat time hardly moves, and whole nodes cut it four to eight times. Shrink the mes­sage to 25 MB and it turns latency-bound, rising with every GPU. The figure's NIC link uses an illus­tra­tive 10 µs, so it shows about 20 ms where the table above, at 5 µs, shows 10 ms. The 25 MB bucket preset uses 5 µs.

Fail­ure modes. "Mis­matched allre­duce order across pro­cesses can lead to wrong results or DDP back­ward hang" [847]. Chunks below the card's 512 KiB buffer go out partly filled [854]. Busbw in a time esti­mate makes an all-reduce look up to twice as fast. How to mea­sure. Run all_reduce_perf, reduce_scatter_perf and all_gather_perf from kilo­bytes to giga­bytes on your job's real groups: the node, the data-par­al­lel group, the whole job [143]. Read busbw at the sizes your frame­work issues, such as DDP's 25 MB buck­ets [855] or one FSDP layer, against 450 and 50 GB/s.

11.2 Data parallelism, ZeRO and FSDP

How do you use many GPUs on one model by split­ting the batch, when does the gra­di­ent traf­fic hide behind com­pute, and how do you stop every GPU from hold­ing the whole train­ing state? Data par­al­lel­ism leaves the model untouched, so it comes first.

DDP and its over­lap

Each rank holds a full replica, runs its slice of the batch, and an all-reduce aver­ages the gra­di­ents before every step, so repli­cas that start equal stay equal [847, 855]. Nor­mal­ize each worker's loss by the global batch, not its own share [366]. The global batch grows with $p$. The crit­i­cal batch size is derived in Sec­tion 5.3, and it caps the useful data-par­al­lel degree.

PyTorch's DistributedDataParallel (DDP) packs gra­di­ents into 25 MB buck­ets in reverse layer order, because col­lec­tives are faster on large ten­sors and gra­di­ents appear from the last layer back. As each bucket fills, a hook launches an asyn­chro­nous all-reduce while ear­lier layers still com­pute, and every rank must launch them in the same order [855]. DDP scales near-lin­early to 256 GPUs [855]. With gra­di­ent accu­mu­la­tion, no_sync() skips all but the last all-reduce, which made ResNet-50 on 256 GPUs 38% faster sync­ing every 8 iter­a­tions [855].

When does gra­di­ent traf­fic hide?

Let a GPU hold $N$ BF16 param­e­ters and process $T_{\text{GPU}}$ tokens between syn­chro­niza­tions at $F$ achieved FLOP/s over a band­width $W$. For­ward costs $2N{T}_{\text{GPU}}$ FLOPs, back­ward $4N{T}_{\text{GPU}}$ (Chap­ter 1). DDP sends about $4N$ bytes per GPU behind the back­ward, so it hides if $4N/W\le 4N{T}_{\text{GPU}}/F$. FSDP all-gath­ers $2N$ bytes behind the for­ward, then all-gath­ers and reduce-scat­ters $4N$ behind the back­ward. All cases give one con­di­tion, inde­pen­dent of model size [856, 138]:

Mul­ti­ply by 1.5 for FSDP with an FP32 reduce-scat­ter and by 2 for DDP with an FP32 all-reduce. The window is one micro-step for FSDP, which regath­ers weights every micro-batch, and one opti­mizer step for DDP with accu­mu­la­tion. Llama 3 reduce-scat­ters in FP32, the 1.5 case [6]. For $W$, use the GPU's egress when shard­ing within a node and the node's when shard­ing across nodes [138].

The whole-node row at peak is the scal­ing book's "about 2500 tokens" per GPU (2,474 by our count), so pure data par­al­lel­ism over 16,384 GPUs would need $\mathrm{16,384}\times \mathrm{2,474}=40.5$M tokens per batch, against Llama 3's 16M [138, 6]. Model par­al­lel­ism shrinks the prob­lem. A GPU in a tensor and pipe­line group holds $N/(p_{\text{tp}}{p}_{\text{pp}})$ param­e­ters but sees all its replica's tokens, $16\times \mathrm{8,192}=\mathrm{131,072}$ per step for each Llama 3 405B replica (com­puted) [6]. The margin still shows in Llama 3's MFU at 16K GPUs (Sec­tion 11.7). And the first all-gather is exposed unless, as in MegaScale (ByteDance's system for train­ing on more than 10,000 GPUs), it is prefetched during data load­ing [47].

ZeRO: shard­ing the train­ing state

Plain data par­al­lel­ism stores all 16 bytes per param­e­ter on every GPU: 2 for BF16 weights, 2 for gra­di­ents and $K=12$ for FP32 master weights and Adam moments [22, 51]. ZeRO shards the opti­mizer state (stage 1), then the gra­di­ents too (stage 2), then the param­e­ters (stage 3) [22]. The bytes per GPU for each stage are derived in Sec­tion 1.4: $4N+KN/p_{\text{dp}}$, $2N+(2+K)N/p_{\text{dp}}$ and $(4+K)N/p_{\text{dp}}$. Here we add what each stage sends. Stages 1 and 2 reduce-scat­ter gra­di­ents and all-gather updated weights, the same $2N$ ele­ments per step as plain data par­al­lel­ism. Stage 3 gath­ers each layer's weights for for­ward, again for back­ward, then reduce-scat­ters: $3N$, 1.5 times the base­line [22].

Stages 1 and 2 never fall below $4N$ and $2N$, 282 and 141 GB for Llama 3 70B, so they cannot fit it on an 80 GB GPU at any degree (com­puted). In Figure 59, pick the 70B preset and see which lines cross the 80 GB mark: only ZeRO-3, and the read­out shows the 1.5 times traf­fic it pays.

Always state dtypes: Mega­tron's dis­trib­uted opti­mizer counts 18 bytes for BF16 param­e­ters with FP32 gra­di­ents [857]. Pipelined runs use stage 1, as in Mega­tron, DeepSeek-V3 and Kimi K2 [857, 2, 72]. Our read­ing of why: a pipelined rank accu­mu­lates gra­di­ents over many micro-batches, and shard­ing them would cost a reduce-scat­ter per micro-batch. Stage 3, as FSDP, is TorchTitan's default and Llama 3's outer axis [48, 6]. Rel­a­tives trade other resources: ZeRO-Off­load uses CPU memory [61], ZeRO-Infin­ity adds NVMe [62] and ZeRO++ quan­tizes stage 3's cross-node traf­fic to a quar­ter [858].

PyTorch FSDP, FSDP2 and HSDP

Fully Sharded Data Par­al­lel (FSDP) is PyTorch's ZeRO-3. It wraps the model in units, usu­ally one block each, and its memory is "pro­por­tional to the size of the sharded model plus the size of the largest fully-mate­ri­al­ized FSDP unit" [859]. Its knobs carry mea­sured effects.

At 175B param­e­ters FSDP reached 173 and 186 TFLOP/s per A100 [859], 55 to 60% of the 312 TFLOP/s peak (com­puted). Flat shard­ing breaks opti­miz­ers that need a whole tensor, such as norm-based or approx­i­mate second-order meth­ods [859] (Chap­ter 5). Hybrid shard­ing (HSDP) shards inside groups of $G$ GPUs and repli­cates across them [860]. Each GPU then all-reduces only its $1/G$ gra­di­ent shard across the $R=p/G$ repli­cas, send­ing for an $n$-byte model

For Llama 3 70B on 1,024 GPUs in groups of 8 that is 35.0 GB, against 281.9 GB for full repli­ca­tion and 422.9 GB for full shard­ing when every byte crosses the net­work (com­puted, the FSDP paper's own for­mula gives 35.2 GB [859]). Hier­ar­chi­cal col­lec­tives cut full shard­ing's bytes per card eight­fold too, so HSDP's last­ing gains are smaller groups and all-gath­ers on NVLink. Its cost is memory: groups of 8 leave 141 GB of 70B state per GPU, groups of 64 leave 17.6 GB (com­puted).

Where FSDP stops scal­ing

This is con­tested. TorchTitan calls FSDP suf­fi­cient "up to 512 GPUs" [48], and the Ultra-Scale Play­book caps data par­al­lel­ism at 512, past which ring latency binds [44]. TorchTitan's Llama 3.1 8B lost 9.8% per GPU from 8 to 128 GPUs (6,258 to 5,645 tokens per second, com­puted from reported values) [48]. On TPUs, How to Scale Your Model spreads Llama 3 70B's 16M-token batch over about 18,800 v5p chips with FSDP alone, assum­ing three inter­con­nect axes [856]. The fab­rics differ in latency: an 80-layer model's 240 per-layer col­lec­tives per micro-step cost 1.2 to 2.5 s of ring latency at 1,024 ranks for $\alpha$ of 5 to 10 µs (illus­tra­tive, com­puted). 512 is a GPU heuris­tic, not a law.

Fail­ure modes. A rank that skips a col­lec­tive hangs DDP [847]. FSDP units under the 33M-ele­ment knee waste band­width, and BF16 gra­di­ent reduc­tion can drift from FP32 [6]. How to do it your­self. Use DDP while the train­ing state fits on one GPU, oth­er­wise FSDP with one unit per block, prefetch­ing and FP32 reduc­tion. Com­pute Table 11.4 for the band­width your group really sees before choos­ing a micro-batch. Past a few hun­dred GPUs, use HSDP or tensor par­al­lel­ism inside the node (Sec­tion 11.3), and pro­file before tuning bucket_cap_mb.

11.3 Tensor and sequence parallelism

How do you split one trans­former layer across GPUs so the math stays exact, and why must that split stay inside an NVLink domain? Once a layer's weights, or one sequence's acti­va­tions, crowd an 80 GB GPU, copy­ing the model to every GPU stops work­ing and you have to cut the layer itself. Tensor par­al­lel­ism (TP) splits each weight matrix across $p_{\text{tp}}$ GPUs. The price is a col­lec­tive inside every layer, so the link decides whether it pays.

Column, then row

Split a weight matrix by col­umns and each GPU com­putes a slice of the output with no com­mu­ni­ca­tion. Split it by rows and each GPU com­putes a par­tial sum that must be added up. Mega­tron-LM chains the two: column-split the MLP's first matrix, apply the non­lin­ear­ity locally, row-split the second matrix so it con­sumes the slices directly, then sum with one all-reduce. In the paper's words, "$f$ is an iden­tity oper­a­tor in the for­ward pass and all reduce in the back­ward pass while $g$ is an all reduce in the for­ward pass and iden­tity in the back­ward pass" [392]. Atten­tion splits the same way by heads, which gives "only two all-reduces in the for­ward path and two in the back­ward path" per layer [392]. SwiGLU fits too, with gate and up pro­jec­tions column-par­al­lel and the down pro­jec­tion row-par­al­lel. Every Llama 3 model has 8 key-value heads [6], so TP 8 leaves one per GPU and any higher degree must repli­cate the key and value pro­jec­tions (derived).

The ratio that decides TP

Each all-reduce car­ries the $b\times s\times d$ acti­va­tion (micro-batch $b$ in sequences, sequence length $s$, width $d$). With the ring cost of Sec­tion 11.1, that is "a total com­mu­ni­ca­tion of $8bsh\left(\frac{t-1}{t}\right)$ per layer per device for each micro­batch", where Narayanan et al. write $h$ for width and $t$ for the TP degree [41]. In BF16 that is $16bsd(p_{\text{tp}}-1)/p_{\text{tp}}$ bytes. A stan­dard layer does $24bs{d}^2$ for­ward FLOPs [41], three times that with the back­ward, shared by $p_{\text{tp}}$ GPUs. With $F$ the peak FLOP/s of one GPU, $W$ its band­width per direc­tion and no over­lap,

Batch and sequence length cancel, so bigger micro-batches cannot rescue TP. Width and the hard­ware ratio $F/W$ decide. An H100 peaks at 989.5 TFLOP/s dense BF16 with 450 GB/s of NVLink per direc­tion [4] and a 50 GB/s NIC per GPU [137], so $F/W$ is about 2,199 FLOPs per byte inside the node and 19,790 across it (com­puted). For $d=\mathrm{8,192}$ the ratio is $0.060(p_{\text{tp}}-1)$ over NVLink, 0.42 at TP 8 for a stan­dard layer, and Llama 3 70B's real layer (grouped-query atten­tion, GQA, and the SwiGLU MLP of Sec­tion 4.2, with causal atten­tion at 8,192 tokens) gives 0.36 (com­puted). Table 11.7 covers the run­ning exam­ples, and the TP pre­sets of Figure 58 let you vary them.

Other sources agree. How to Scale Your Model finds TP com­mu­ni­ca­tion-bound beyond 8 to 16 ways for most models on TPU v5p [856]. The Ultra-Scale Play­book mea­sured "sig­nif­i­cant drops when moving from TP=8 to TP=16", and about 43% lost when TP crosses nodes against 14% for pipe­line par­al­lel­ism [44]. Narayanan et al.'s rule is TP "up to degree $g$ when using $g$-GPU servers", with pipe­line par­al­lel­ism beyond [41].

Sequence par­al­lel­ism and acti­va­tion memory

Between TP's col­lec­tives, norms and resid­ual adds run on the full, repli­cated tensor. Sequence par­al­lel­ism (SP) splits them along $s$, turn­ing each all-reduce into a reduce-scat­ter and an all-gather of the same total size, so it "does not intro­duce any com­mu­ni­ca­tion over­head" [45]. Here "sequence par­al­lel­ism" means only this com­pan­ion of TP. Ring Atten­tion and Ulysses are con­text par­al­lel­ism (Sec­tion 11.5).

Acti­va­tion memory per layer is derived in Sec­tion 1.5. For a GPT layer it is $sbd(34+5hs/d)$ bytes with $h$ heads [45]. Table 11.8 divides it for GPT-3 175B, whose $5hs/d=80$ term is more than twice the 34, which is why recom­put­ing only the atten­tion scores (selec­tive recom­pu­ta­tion) pays.

Over­lap helps at the margin. PyTorch's asyn­chro­nous TP splits the all-gather and its matrix prod­uct into pieces and reports about 8% end-to-end speedups for 8B- and 70B-class Llama models, with the catch that copy ker­nels take stream­ing mul­ti­pro­ces­sors away from the matrix prod­ucts (reported) [861, 862]. To do it your­self: set TP to one NVLink domain (8 on an H100 server), with one tech­nique across 8 GPUs often enough under about 10B param­e­ters [44], always add SP, check that heads, MLP width and vocab­u­lary divide by $p_{\text{tp}}$, and com­pare the pro­filed col­lec­tive time per layer with Table 11.7.

11.4 Pipeline parallelism and its schedules

Tensor par­al­lel­ism is stuck inside a node: how do you split a 126-layer model across nodes when the net­work is about ten times slower, and how much idle time does that cost? Cut the stack into $p_{\text{pp}}$ stages and pass acti­va­tions for­ward and gra­di­ents back­ward at the cuts. Each cut car­ries one tensor per micro-batch, point to point, so "from a com­mu­ni­ca­tion stand­point, pipe­lin­ing is basi­cally free" [138]. The cost is idle time, the bubble.

The bubble, derived

Split the batch into $m$ micro-batches with for­ward and back­ward times $t_f$ and $t_b$ per stage. If every stage runs all for­wards then all back­wards, as GPipe does [863], "the pipe­line bubble con­sists of $p-1$ for­ward passes at the start of a batch, and $p-1$ back­ward passes at the end" [41]. Each GPU idles $(p_{\text{pp}}-1)(t_f+t_b)$ while work­ing $m(t_f+t_b)$. With $v$ vir­tual stages per GPU (below),

Narayanan et al. and Llama 3 use the first form, which this guide fol­lows, and GPipe the second [863, 6]. GPipe found the bubble neg­li­gi­ble once the micro-batches number at least four times the stages ($m\ge 4p_{\text{pp}}$), but it holds all $m$ micro-batches' acti­va­tions [863]. PipeDream's one-for­ward-one-back­ward (1F1B) order [864], in its syn­chro­nous form [865], caps the micro-batches in flight at the pipe­line depth with the same bubble [41]. 1F1B fixes memory, which lets you raise $m$. Inter­leav­ing gives each GPU $v$ non-con­tigu­ous chunks of layers, which "reduces the bubble time by $v$", mul­ti­plies point-to-point traf­fic by $v$, requires $m$ to be a mul­ti­ple of $p_{\text{pp}}$ and gained up to 10% through­put (reported) [41].

Zero bubble and DualPipe: split the back­ward

A linear layer's back­ward is two prod­ucts: the input gra­di­ent, needed by the pre­vi­ous stage now, and the weight gra­di­ent, needed only by the opti­mizer. With times $T_F$ (for­ward), $T_B$ (input gra­di­ent) and $T_W$ (weight gra­di­ent), "W can be flex­i­bly sched­uled any­where after the cor­re­spond­ing B of the same stage", and $T_B+T_W=2T_F$ under a FLOP model [866]. Fill­ing holes with W passes gives bub­bles per GPU of

ZB-H1 keeps 1F1B's memory and cuts its bubble to a third when the three times are equal. An auto­matic sched­uler gets under 1% in most set­tings and up to 23% more through­put than 1F1B at sim­i­lar memory, though the opti­mizer step's syn­chro­niza­tion "makes zero bubble impos­si­ble" unless worked around (reported) [866]. DeepSeek-V3's DualPipe feeds micro-batches from both ends, holds two param­e­ter copies and pairs a for­ward chunk with a back­ward chunk, $T_{F\&B}$ above, so one's expert all-to-all hides behind the other's com­pute. Cross-node expert par­al­lel­ism had left "an inef­fi­cient com­pu­ta­tion-to-com­mu­ni­ca­tion ratio of approx­i­mately 1:1" [2]. The report writes $B$ for the full back­ward, $T_B+T_W$ here, so check which $B$ a paper means.

Figure 60 runs five sched­ules on four stages and eight micro-batches. Step through GPipe and 1F1B first: both take 33 units with 9 idle per GPU (27.3% of the step), but 1F1B holds at most 4 micro-batches against GPipe's 8. Then watch inter­leav­ing, ZB-H1 and DualPipe finish at 28.5, 27 and 26 units, 15.8, 11.1 and 7.7% idle (com­puted). The sim­u­la­tion gives DualPipe no credit for hiding com­mu­ni­ca­tion. The cal­cu­la­tor below the time­lines com­putes $(p_{\text{pp}}-1)/(mv)$ for your own run.

Memory: the first stage holds the most

Pipe­lin­ing changes how many layers' worth of acti­va­tions (Sec­tion 1.5) a stage holds: under 1F1B, "the first stage must store $p\times L/p=L$ layers worth of acti­va­tions regard­less of the pipe­line par­al­lel size $p$", and inter­leav­ing mul­ti­plies that by $1+(p_{\text{pp}}-1)/(p_{\text{pp}}v)$ [45]. Pipe­lin­ing divides weights, not the first stage's acti­va­tions. Llama 3 found its first stage heavy from "the embed­ding and the warm-up micro-batches", and removed one trans­former layer each from the first and last stages [6], so $126+2=128=16\times 8$ slots split evenly for $v$ of 1, 2, 4 or 8 (com­puted).

Watch for a slow stage pacing the rest, micro-batch counts that fight batch-size warmup, exposed point-to-point latency with tiny micro-batches, and drift when many micro-batches accu­mu­late in low pre­ci­sion (Llama 3 accu­mu­lates in FP32 [6]). To do it your­self, keep the pipe­line degree as small as memory allows, use $m\ge 4p_{\text{pp}}$ or inter­leave, check the first stage's memory rather than the aver­age, and com­pute the bubble before you launch.

11.5 Context parallelism for long sequences

At 128K tokens even one sequence's acti­va­tions over­flow a tensor-par­al­lel group: how do you split atten­tion, where every query needs every ear­lier key? Every­thing out­side atten­tion treats tokens inde­pen­dently and splits along the sequence for free. Con­text par­al­lel­ism (CP) gives each of $p_{\text{cp}}$ GPUs a slice of the sequence and either moves keys and values to the queries (Ring Atten­tion, all-gather CP) or reshards from sequence to heads (Ulysses). Long-con­text recipes are in Sec­tion 7.5.

Ring Atten­tion and all-gather CP

In Ring Atten­tion each GPU keeps its queries while key and value blocks travel a ring, and com­putes block­wise atten­tion on one block while the next arrives [867, 868]. A block of $c$ tokens costs about $4d{c}^2$ FLOPs and $4cd$ bytes, so the trans­fer hides when $c\ge F/W$ [867]. Grouped-query atten­tion shrinks keys and values to width $d_{kv}=h_{kv}{d}_h$, so

For Llama 3 70B, $d_{kv}/d=1/8$, so on an H100 at peak the block must be about 275 tokens over NVLink and 2,474 over a NIC, against 2,199 and 19,790 with full multi-head atten­tion (com­puted) [867, 4]. GQA makes ring CP across nodes prac­ti­cal. Causal masks unbal­ance con­tigu­ous slices, so Striped Atten­tion spreads each GPU's tokens through the sequence for up to 1.45 times Ring Atten­tion's through­put at 256K [869], and Llama 3's zigzag split gives rank $i$ chunks $i$ and $2p_{\text{cp}}-1-i$ [6].

Llama 3 skips the ring and all-gath­ers keys and values once per layer. All-gath­ers sup­port masks such as the doc­u­ment mask more easily, and "the exposed all-gather latency is small as the com­mu­ni­cated K and V ten­sors are much smaller than Q tensor due to the use of GQA" [6]. Each GPU receives $\frac{p_{\text{cp}}-1}{p_{\text{cp}}}·2s{d}_{kv,\text{local}}·2$ bytes per layer, with $d_{kv,\text{local}}$ the key-value width left after TP.

Ulysses, and how to choose

DeepSpeed-Ulysses uses an all-to-all so each GPU trades its $s/p_{\text{cp}}$ tokens with all heads for all $s$ tokens with $h/p_{\text{cp}}$ heads, runs ordi­nary atten­tion, and swaps back. Its per-link volume is $p_{\text{cp}}$ times smaller than Mega­tron SP's [870]. The limit is heads: "Llama3⁠–⁠8B employs GQA with a KV head number of 8, which means that when using DS-Ulysses SP, the max­i­mum SP degree is 8", and Ulysses com­petes with TP for the same heads [871]. USP com­bines Ulysses and a ring on a 2D grid and reached 47% MFU for Llama 3 8B at 208K tokens on two 8-GPU A800 nodes (reported) [871].

With GQA, ring and all-gather CP move the same 60 MiB, and Ulysses cannot reach degree 16 at TP 8 with­out repli­cat­ing K and V, which roughly dou­bles its bytes. So choose by over­lap, masks and heads. Newer sys­tems handle packed doc­u­ments of mixed length with dynamic meshes, ByteScale report­ing up to 7.89 times the prior state of the art on more than 12,000 GPUs [872], and the same ideas power mil­lion-token pre­fill at infer­ence [873] (Chap­ter 13). To do it your­self, turn CP on only for long sequences, as Llama 3 did (CP 1 at 8K, 16 at 128K) [6], keep the group inside a pod, check the block con­di­tion with your $d_{kv}/d$, bal­ance causal masks with zigzag or striped assign­ment, and pick all-gather CP for doc­u­ment masks and Ulysses when key-value heads are plen­ti­ful. Sec­tion 11.7 puts CP among the other axes.

11.6 Expert parallelism and the all-to-all

A mix­ture-of-experts layer has many expert MLPs and each token uses only $k$ of them: where do the experts live, and how do tokens reach them? Each DeepSeek-V3 MoE layer has 1 shared and 256 routed experts of width 2,048, and each token uses 8 routed experts on at most 4 nodes [2]. Rout­ing and bal­anc­ing are in Sec­tion 4.7.

The axes so far split the work of every token. Expert par­al­lel­ism (EP) splits the set of experts instead: each GPU holds whole experts and tokens travel to them, so noth­ing inside an expert's matrix mul­ti­ply needs a col­lec­tive [44]. The price is an all-to-all (Sec­tion 11.1) in every MoE layer.

The gra­di­ent of a dis­patch is a com­bine, so one MoE layer costs four all-to-alls per micro-batch. EP groups are carved out of the data-par­al­lel group: dense weights are data-par­al­lel over the whole group, expert weights sharded across the EP sub-group. MoE Par­al­lel Fold­ing removes the old rule that the EP degree cannot exceed the data-par­al­lel degree, by giving atten­tion and experts sep­a­rate meshes, at 49.3% MFU on Mix­tral 8x22B [49].

How many bytes, and when EP beats TP

With $p_{\text{ep}}$ GPUs, $T$ tokens per GPU, width $d$, $b_e$ bytes per ele­ment and uni­form rout­ing, one dis­patch sends

bytes per GPU, and the com­bine moves as many ele­ments. For $k=8$, $d=\mathrm{7,168}$, FP8 and an illus­tra­tive $T=\mathrm{4,096}$: $\mathrm{4,096}\times 8\times \mathrm{7,168}\times 7/8=206$ MB at $p_{\text{ep}}=8$ and 231 MB at 64 (com­puted). The volume barely depends on EP size, only the link it crosses does. Against TP on the same $p_{\text{ep}}$ GPUs, which all-reduces every token's acti­va­tion, EP's dis­patch plus com­bine moves $k/p_{\text{ep}}$ as much. For $p_{\text{ep}}=8$, $k=2$ and 32,768 BF16 tokens (4,096 per GPU), dis­patch and com­bine send 103 + 103 = 206 MB per GPU, against 822 MB for TP's ring all-reduce of the 470 MB acti­va­tion (com­puted). MegaScale-MoE switches to all-gather and reduce-scat­ter when $k>p_{\text{ep}}$, "ensur­ing that EP’s com­mu­ni­ca­tion over­head remains equal to or lower than TP’s" [50].

Repli­cat­ing experts with data par­al­lel­ism instead mul­ti­plies the break-even of Sec­tion 11.2 by $E/k$, since gra­di­ents scale with all $E$ experts and com­pute with $k$:

For $E=128$, $k=4$ that is "32 * 2475 = 79,200 across nodes, a kind of ridicu­lously high number" [138]. So MoE runs shard their experts.

DeepSeek-V3: cross InfiniBand once per node

DeepSeek-V3 trained on 2,048 H800s with 16-way pipe­line par­al­lel­ism, 64-way EP over 8 nodes and ZeRO-1, "with­out using costly Tensor Par­al­lel­ism (TP)" [2], so each GPU holds $256/64=4$ routed experts per layer (com­puted). The H800's NVLink is cut to 400 GB/s in total, 200 GB/s each way against the H100's 450 [1, 4], and DeepSeek mea­sures 160 GB/s over NVLink against 50 GB/s over InfiniBand (IB) [2]. Each des­ti­na­tion node gets a token once over IB and fans it out over NVLink, so IB car­ries $q$ copies for $q$ nodes [1], with $q\le 4$. Dis­patch goes in FP8 and the com­bine in BF16 [2], so IB bytes per token per MoE layer in the for­ward pass are $V_{\text{IB}}=qd(1+2)$.

Fail­ure modes: imbal­ance becomes strag­glers, dropped tokens cost qual­ity silently, and com­mu­ni­ca­tion ker­nels steal SMs. How to do it your­self: keep EP in the fastest domain, cross nodes only with a node limit, prefer EP to TP for experts [138], and budget SMs.

11.7 Composing the axes: meshes, real runs and rules of thumb

Given a model, a batch and a clus­ter, which degree do you give each axis and which GPUs share a group, and what did the largest pub­lished runs actu­ally do? The rule from the open­ing holds: the chat­ti­est axis gets the fat­test pipe. This sec­tion turns it into degrees and groups.

Each GPU gets a coor­di­nate on a grid with one dimen­sion per axis. Llama 3 forms groups "in the order of [TP, CP, PP, DP], where DP stands for FSDP", so TP groups are con­sec­u­tive ranks in one server and DP, which "can tol­er­ate longer net­work latency", is out­er­most [6]. In code this grid is a DeviceMesh or a JAX Mesh (Sec­tion 11.9). With EP folded into DP and $b_{\text{replica}}$ sequences per DP group, two checks follow:

Llama 3's mesh sits on 8-GPU NVLink servers, 3,072-GPU pods with full bisec­tion band­width, and a 1:7 over­sub­scribed layer above the pods [6] (Sec­tion 2.6). A 16,384-GPU job spans 5.33 pods (com­puted), so only the out­er­most axis should cross that layer often.

Llama 3's three stages [6] each mul­ti­ply to their GPU count, and the tokens per step agree (com­puted):

Dou­bling the GPUs halved the sequences per DP group, and the paper blames the drop from 43% to 41% on that "lower batch size per DP group" [6]: fewer micro-batches, a larger bubble. In Figure 61, hover a GPU to light up its server (TP row), its 16-server pipe­line (PP column) and its FSDP peers in the other repli­cas, and switch to the 128K stage to watch CP 16 replace most of DP.

Why not plain FSDP over all 16,384 GPUs? Pure data par­al­lel­ism would need about 40.5M tokens per batch (16,384 times the 2,474 of Table 11.4, com­puted), the scal­ing book's "at least 40M" [138], and the quiz at the end works through why. The same book calls Llama 3's DP "128-way ZeRO-1" [138], but the paper's FSDP "shards opti­mizer states and gra­di­ents" and keeps weights between for­ward and back­ward [6]. Its weights stay sharded between steps but are not resharded after the for­ward pass, which is ZeRO-3 stor­age run like ZeRO-2 within a step (our read­ing). Prefer the pri­mary source to the sum­mary.

Axes also trade: bal­anc­ing FSDP's weight traf­fic against TP's acti­va­tion traf­fic gives the opti­mal FSDP degree $X_{\text{opt}}=\sqrt{(B/d_{\text{ff}})(M_X/M_Y)p}$ (the book's $M_X$ and $M_Y$ count the hard­ware mesh axes each strat­egy spans) [856], 13.7 for $B=\mathrm{48,000}$, $d_{\text{ff}}=\mathrm{32,768}$, $p=64$ and a ratio of 2 (com­puted, the book's 13.9).

Treat these as priors: in the Play­book's tests TP beat PP until the team opti­mized its PP code [44].

Real runs, and how they count

Figure 62 plots pub­lished runs: filled dots for reported MFU, hollow for num­bers count­ing recom­pu­ta­tion or masked atten­tion. Reported MFU spans 28% (MegaScale-MoE at 1,440 GPUs) to 55% (MegaScale's 175B) [50, 47].

MFU and its con­ven­tions (6N or with atten­tion, which peak, which time base) are defined in Sec­tion 1.3. Say which one a number uses. Mega­tron's 1T run reported "52% of the­o­ret­i­cal peak" count­ing recom­pu­ta­tion [41], about $52.4\%\times 3/4=39\%$ as MFU (com­puted). PaLM's 46.2% MFU is 57.8% as hard­ware FLOPs uti­liza­tion [34], and DeepSeek-V3's 38.9% is 43.7% if atten­tion counts the full matrix [1]. Pub­lish tokens per second per GPU, FLOPs per token and the peak. In the figure's second view, MegaScale's abla­tion climbs from 47.7% to 65.3%, and over­lap­ping TP, PP and DP com­mu­ni­ca­tion buys 6.2 points, more than its two model changes together [47].

Fail­ure modes: a poor TP and PP split costs up to 2 times in through­put [41], too few sequences per DP replica inflate the bubble, and lay­outs drift numer­i­cally. How to do it your­self: make TP groups con­tigu­ous ranks in a server, check the degree prod­uct and tokens per step, and com­pare loss curves across lay­outs.

11.8 Training across data centers: low-communication methods

Can one model be trained across data cen­ters, or over the inter­net, where band­width is 100 to 1,000 times lower than InfiniBand? Not syn­chronously. At 1 Gbit/s the break-even of Sec­tion 11.2 for an H100 at 40% MFU is $0.4\times 989.5\times {10}^{12}/(0.125\times {10}^9)\approx 3.2$M tokens per GPU per step (com­puted). INTELLECT-1 saw 500 Mb/s to 4 Gb/s between nodes [877], while data par­al­lel­ism needs 100 to 300 Gbit/s for 1B to 100B models at 95% uti­liza­tion (sim­u­lated) [878]. The levers are to syn­chro­nize less often, send fewer bits and over­lap, start­ing from local SGD [879, 880, 881].

DiLoCo runs $R$ repli­cas, each taking $H$ inner AdamW steps with its own Adam state, then aver­ages the param­e­ter changes with one all-reduce and applies an outer Nes­terov step [882]:

# DiLoCo, one outer round (after Douillard et al., 2023)
for r in range(R):                         # in parallel, no communication
    theta_r = theta.copy()
    for h in range(H):
        theta_r = adamw_step(theta_r, next_batch(r))
    delta[r] = theta - theta_r             # outer gradient
theta = nesterov_step(theta, all_reduce_mean(delta))  # 1 sync / H steps

DiLoCo beats data par­al­lel­ism at equal com­pute while "com­mu­ni­cat­ing 500 times less", but loses to 8 times more sequen­tial updates, so it "reduces the FLOP and data effi­ciency of the model" [882]. Whether it beats syn­chro­nous train­ing depends on what is held fixed. Scal­ing laws say the 4-replica loss penalty falls from 2% at 35M param­e­ters to 0.26% at 2.4B [883], while a 2026 paper notes that the gap grows with repli­cas and local steps and nar­rows it with a merg­ing-style aggre­ga­tion, IsoLoCo [884].

Two cau­tions. NoLoCo's claim that global syn­chro­niza­tion "scales quadrat­i­cally with group size" [888] holds for the pair­wise traf­fic of naive aver­ag­ing, not for a ring all-reduce, whose bytes per GPU are flat in $p$ (Sec­tion 11.1). And no fron­tier report in this chap­ter com­presses gra­di­ents: Llama 3, DeepSeek-V3 and MegaScale reduce in BF16 or FP32 over fast fab­rics [6, 2, 47].

11.9 Frameworks, compilers and choosing a configuration

Which soft­ware imple­ments all of this, can a com­piler choose the layout for you, and how do you pick a con­fig­u­ra­tion with­out trying every one? Com­pil­ers derive col­lec­tives from a few shard­ing hints. GSPMD "always pro­duces a valid par­ti­tioned graph regard­less of what shard­ing anno­ta­tions are pro­vided" and reached 50⁠–⁠62% uti­liza­tion on up to 2,048 TPU v3 cores [889]. Alpa picks pipe­line stages by dynamic pro­gram­ming and shard­ing by an inte­ger linear pro­gram, and matches Mega­tron-LM on GPT [890], after FlexFlow's ear­lier search [891]. JAX has both styles: jit shards auto­mat­i­cally, while "With shard_map you take con­trol" [892]. PyTorch stacks its axes "through DeviceMesh and DTensor" with­out a whole-pro­gram par­ti­tioner [893]. Pipe­lin­ing fits auto­matic par­ti­tion­ers poorly [138], and DualPipe is the hand-writ­ten extreme (Sec­tion 11.4).

Our opin­ion: learn on Nan­otron or TorchTitan, whose code is short. Use Mega­tron Core for the largest NVIDIA runs, MaxText or Lev­an­ter on TPUs, and DeepSpeed's off­load on one memory-starved node.

Pre­dict before you search. Narayanan et al.'s esti­mate $t_{\text{train}}\approx 8DN/(pF)$ [41], checked against GPT-3's 34 days in Sec­tion 1.3, is the first filter. Llama 3 built "a memory con­sump­tion esti­ma­tor and a per­for­mance-pro­jec­tion tool" [6]. The Play­book's brute force took over 4,100 exper­i­ments, and "Some jobs would hang indef­i­nitely" [44]. Our toy search for Llama 3 70B on 1,024 H100s (illus­tra­tive: 4M-token batch, 18 bytes per param­e­ter, 60% matmul effi­ciency) put the best TP 4 lay­outs at 49⁠–⁠52% model MFU and TP 8 at 48.8%. Gaps that small are why teams bench­mark the final­ists.

Why does ring all-reduce time barely change from 64 to 1,024 GPUs for a 1 GB mes­sage, but grow about 15-fold for 1 MB?

The band­width factor $2(p-1)/p$ moves only from 1.97 to 2.00, while the latency term $2(p-1)\alpha$ grows 16 times. At 50 GB/s, 1 GB costs about 40 ms of band­width time, 1 MB only 0.04 ms, while latency at an illus­tra­tive 5 µs grows from 0.63 to 10.2 ms (com­puted). Trees fix that, since ring latency "scales lin­early with the number of GPUs" [852].

A 13B model with mixed-pre­ci­sion Adam on 32 GPUs: which ZeRO stage is the prac­ti­cal min­i­mum in 80 GB?

At 16 bytes per param­e­ter (BF16 weights and gra­di­ents, FP32 master weights and Adam moments), plain DP needs 208 GB. ZeRO-1 needs $52+156/32=56.9$ GB, leav­ing little for acti­va­tions. ZeRO-2 needs $26+182/32=31.7$ GB at no extra com­mu­ni­ca­tion. So ZeRO-2 is the prac­ti­cal min­i­mum. ZeRO-3 needs 6.5 GB for 1.5 times the bytes (com­puted) [22].

Why does a bigger micro-batch not cut tensor par­al­lel­ism's com­mu­ni­ca­tion share?

TP's bytes and FLOPs both scale with $b\times s$ [41], so the ratio $\frac{2}{9}(p_{\text{tp}}-1)F/(dW)$ depends only on the degree, the width and the hard­ware. Bigger micro-batches help the bubble and ker­nels. Only a faster link helps TP.

With 8 stages and 16 micro-batches, what bubble do 1F1B and inter­leav­ing with v = 4 give?

1F1B: $7/16=0.44$ of ideal time, 30% of the step. Inter­leaved: $7/64=0.11$, 10% (com­puted). Inter­leav­ing mul­ti­plies point-to-point traf­fic by $v$ and needs $m$ to be a mul­ti­ple of $p_{\text{pp}}$ [41].

Why did Llama 3 405B not run plain FSDP over all 16,384 GPUs?

16M tokens is "about 1k tokens per GPU", below the roughly 2,500 needed to hide data-par­al­lel traf­fic across nodes [138]. With TP 8 and PP 16 each replica pro­cesses 131,072 tokens, far above that, while each GPU holds 1/128 of the weights. The 6.49 TB of train­ing state could not sit unsharded anyway (com­puted).

12. Running the run: reliability, observability and ML operations

A large run can lose progress by stop­ping, slow­ing down or con­tin­u­ing with­out learn­ing what it should. Check­points and recov­ery pro­tect the work already paid for; observ­abil­ity and repro­ducibil­ity help estab­lish that the work is still useful.

A run on 16,384 GPUs burns 16,384 GPU-hours every wall-clock hour and stops every few hours: how does it ever finish, and how do you know it is learn­ing what you think? Stand­ing still is expen­sive: the OPT-175B log­book warned that idle nodes cost $2,500 an hour [901], about $2.5 per GPU-hour on its 992 GPUs [345] (com­puted). It is also common. In a 54-day snap­shot, Llama 3 405B pre­train­ing was inter­rupted 466 times, 47 planned and 419 unex­pected, one unex­pected stop every 3.1 hours (com­puted). Yet the team reports "higher than 90% effec­tive train­ing time", with sig­nif­i­cant manual inter­ven­tion only three times [6].

Think of the run as a long-lived dis­trib­uted ser­vice whose mean time to fail­ure falls as one over its size. The chap­ter fol­lows the loop that keeps it up: fail­ures (Sec­tions 12.1 and 12.2), check­points and recov­ery (Sec­tions 12.3 and 12.4), sig­nals, play­books and deter­min­ism (Sec­tions 12.5 to 12.7), then good­put and oper­a­tions (Sec­tions 12.8 and 12.9).

Llama 3 405B is the main exam­ple, DeepSeek-V3 (2,048 GPUs) and GPT-2 124M (one node) mark the ends of the scale axis, and OPT-175B and Gemini are case stud­ies. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. See Chap­ter 11 for why every step needs every rank, Sec­tion 1.3 for MFU and Sec­tion 2.7 for the hard­ware that fails.

12.1 Why big runs stop

A single GPU almost never fails, so why does a 16,384-GPU job stop sev­eral times a day, and what actu­ally breaks? Because syn­chro­nous train­ing is a series system: every step needs every rank, so one GPU fail­ure can force a restart of the entire job [6]. Gemini's report puts it this way: "Scal­ing up the number of accel­er­a­tors results in a pro­por­tion­ate decrease in the mean time between fail­ure of hard­ware in the over­all system." [902]

Let each node fail inde­pen­dently as a Pois­son process with $r_f$ fail­ures per node-day (the fail­ure rate, quoted per 1,000 node-days). A node sur­vives time $t$ with prob­a­bil­ity $e^{-r_{f}t}$, so all $n_{\text{nodes}}$ sur­vive with prob­a­bil­ity $e^{-n_{\text{nodes}}{r}_{f}t}$. That is an expo­nen­tial life­time with rate $n_{\text{nodes}}{r}_f$, and the job's mean time to fail­ure (MTTF) is

where the second form esti­mates the rate from a run log. Koko­lis et al. use the same model [903]. Dou­bling the job halves $M$.

What the mea­sure­ments say

Over 11 months and 150 mil­lion A100 GPU-hours on Meta's research clus­ters, Koko­lis et al. mea­sured 6.50 fail­ures per 1,000 node-days on RSC-1 and 2.34 on RSC-2, on jobs above 128 GPUs. MTTF was 47.7 days for 8-GPU jobs and 7.9 hours at 1,024 GPUs, and they project 1.8 hours at 16,384 GPUs and 0.23 hours at 131,072 (all reported) [903]. The pro­jec­tion is our for­mula: $24/(\mathrm{2,048}\times 0.0065)=1.80$ hours for 2,048 eight-GPU nodes (com­puted).

It is an order-of-mag­ni­tude guide: at 1,024 GPUs RSC-1's rate pre­dicts $24/(128\times 0.0065)=28.8$ hours (com­puted), not the mea­sured 7.9. The authors say the model "matches well with observed MTTF num­bers for larger jobs". The rate also moves, from near 2.5 to spikes around 17.5, and each new health check raises it by count­ing a fail­ure mode that was already there [903]. Table 12.1 places the run­ning exam­ples on the curve. At 131,072 GPUs a 10-minute restart is 27⁠–⁠74% of the MTTF (com­puted, $R/M$), far out­side the regime where the simple model holds.

Run reports imply sim­i­lar rates (com­puted): OPT-175B about 14, from at least 35 manual and an esti­mated 70+ auto­matic restarts in two months on 124 nodes [345], and BLOOM about 4.5, from one to two GPU fail­ures a week on 48 nodes [401]. Figure 63 puts it all on log-log axes. Look for the slope of minus one, the mid-scale mea­sure­ments below the RSC-1 line, and the band where MTTF is under ten restart costs.

What breaks

Llama 3's Table 5 is the only public per-com­po­nent log of a fron­tier run. By its counts, GPU prob­lems (faulty GPUs, HBM3 and SRAM memory, the GPU system pro­ces­sor, silent cor­rup­tion, ther­mal sen­sors) cause 268 of the 419 unex­pected inter­rup­tions, soft­ware and depen­den­cies 56 and the net­work 35 (com­puted from [6]). In Figure 64, note how far faulty GPUs and HBM3 stand out. Then switch to ByteRobust's plat­form table over 778,135 jobs, led by CUDA errors, CPU over­load and memory, hangs and manual restarts [904]. These are fleet inci­dents, not one job's inter­rup­tions. Llama 3's response was faster startup and check­points plus diag­no­sis tools [6], the sub­jects of Sec­tion 12.3, Sec­tion 12.4 and Sec­tion 12.6.

Few jobs, most of the lost time

Fail­ures con­cen­trate in big jobs. At Meta they hit 0.2% of jobs but 18.7% of run­time [903]. On Shang­hai AI Lab's clus­ters, infra­struc­ture fail­ures were 11% of failed jobs but over 82% of failed GPU time [905]. MegaScale restarted over 100 times in sev­eral weeks on over 10,000 GPUs, recov­er­ing over 90% of excep­tions auto­mat­i­cally [47]. Bad nodes repeat: detect­ing them cut fail­ures of 512-plus-GPU jobs from 14% to 4% [903].

In prac­tice, esti­mate $r_f$ from your clus­ter's his­tory and com­pute $M$ for your job. If $M$ is under about ten times your restart cost, invest in restart speed first (a rule of thumb). The model breaks down under cor­re­lated fail­ures, infant mor­tal­ity and chang­ing health checks. Ear­lier HPC stud­ies [906, 907] and recent GPU-fleet stud­ies [908, 909] give back­ground.

Llama 3 405B had 419 unex­pected inter­rup­tions in 54 days on 2,048 eight-GPU nodes. What MTTF should a 2,048-GPU job like DeepSeek-V3's expect at the same rate?

The rate is $419/(\mathrm{2,048}\times 54)=3.79$ per 1,000 node-days (com­puted). MTTF scales as one over the node count. With 8 times fewer nodes it is $8\times 3.09\approx 25$ hours, about a fail­ure a day (com­puted). Treat it as an order of mag­ni­tude, since mea­sured mid-scale MTTFs run lower.

12.2 Failures that do not crash: silent data corruption and stragglers

What if a GPU com­putes a wrong answer with­out report­ing an error, or keeps work­ing but slowly? Nei­ther crashes the job, so nei­ther trips the restart loop of Sec­tion 12.4.

Silent data cor­rup­tion

Silent data cor­rup­tion (SDC) is hard­ware return­ing a wrong result with no error signal, so it is "not trace­able at the hard­ware level" [910]. In one of Meta's cases a power func­tion "would always pro­duce a result of 0 on core 59 of the CPU" for one input [910]. Google sees "a few mer­cu­rial cores per sev­eral thou­sand machines" [911], and test escapes run at least ten times above indus­trial tar­gets [912].

Llama 3 saw one every 9 days (com­puted above). Gemini's team expected SDC "to impact train­ing every week or two" [902]. For Gemini 2.5, around 0.25% of steps were replayed for sus­pected SDC and 6% of those were gen­uine [913], so $0.0025\times 0.06=0.015\%$ of steps were cor­rupted (com­puted).

SDC barely shows. Com­par­ing unhealthy and healthy nodes under deter­min­is­tic exe­cu­tion, Ma et al. found the gra­di­ent noise is "small rel­a­tive to the true gra­di­ent norm", yet param­e­ters drift from ground-truth weights, and SDC "can silently occur with­out any clear indi­ca­tion from train­ing loss" [914]. On DeepSeek's Fire-Flyer clus­ter, errors that ECC missed caused gra­di­ent-norm spikes, loss explo­sions and non-con­ver­gence [915]. Detec­tion is a ladder, cheap­est first:

1. Fleet scans. Meta's Ripple catches 70% of cov­ered errors within 15 days [916], and Fire-Flyer val­i­dates nodes weekly [915].
2. Train­ing sig­nals. Watch­ing for abrupt changes in gra­di­ents or acti­va­tions is the common guard [903], but Ma et al. found SDC that per­turbed gra­di­ents with no clear sign in the loss [914].
3. Deter­min­is­tic replay. Gemini 2.5 repeats "any step with sus­pi­cious met­rics" and com­pares per-device check­sums [913]. After unex­plained NaNs, ByteRobust runs one step on fixed input every­where and removes machines whose out­puts differ bit for bit [904].
4. Check­summed matrix mul­ti­plies missed most of Ma et al.'s unhealthy nodes [914], so as of Novem­ber 2026 treat them as unproven [917, 918].

Sever­ity is con­tested: such events "rarely war­rant user com­plaints" on a research clus­ter [903], but Gemini expects one every week or two. It grows with job size and with whether anyone looks. Keep a bit­wise-deter­min­is­tic con­fig­u­ra­tion (Sec­tion 12.7) and a golden run for sus­pect nodes, and replay a sus­pi­cious step on other hard­ware before rolling back blindly.

Strag­glers

A strag­gler (or fail-slow) is a rank that keeps work­ing, but slowly. A syn­chro­nous step ends when its slow­est rank does, so "Even a single strag­gler can slow down thou­sands of other GPUs" [6]. Llama 3's team found cul­prits by rank­ing sus­pi­cious com­mu­ni­ca­tions [6]. Even pure noise costs time. If $p$ ranks have inde­pen­dent step times with mean $\mu$ and stan­dard devi­a­tion $\sigma$, extreme-value theory gives

which over­es­ti­mates at every $p$: by about 40% of the excess at 8 ranks and about 10% at 16,384 (com­puted).

Hard­ware strag­glers exist too. Remov­ing MegaScale's 0.5% of slow machines gained about 0.7% MFU [47]. Hot GPUs clock down and drag their neigh­bors [921]: in one node Guard mea­sured 1.93 GHz at 50 °C and 1.38 GHz at 77 °C [922], a 28% drop (com­puted). Redun­dancy can hide such slow­ness as gray fail­ure [923].

12.3 Checkpoints: what to save, how to save it, how often

With a fail­ure every three hours, what must a check­point con­tain, how do you write ter­abytes with­out stop­ping 16,384 GPUs, and how often should you do it? Each check­point stalls train­ing briefly, and each fail­ure throws away the work since the last one. A short deriva­tion finds the bal­ance, and shows that once check­points are fast, the restart path mat­ters more.

What goes in, and how big it is

A restartable check­point holds GPU state, mean­ing the param­e­ters, the FP32 master weights and Adam's two moments, and CPU state: the data loader's posi­tion, the random-number-gen­er­a­tor states, the step counter and the learn­ing-rate sched­uler [924]. Leave a piece out and the resumed run repeats data or resumes at the wrong learn­ing rate. In FP16 train­ing the loss-scaler state mat­ters too: OPT-175B's log­book records a resumed run that diverged after its unsaved loss-scale state was reset [901]. Size fol­lows from bytes per param­e­ter, with BF16 weights (2 B), FP32 master weights (4 B) and two FP32 moments (4 B each):

Gra­di­ents are recom­puted after a restart and need not be saved, so 16 B, which adds them, is an upper bound (Sec­tion 1.4). The BF16 copy can be recast from the FP32 master weights, so 12 B is the min­i­mum and 14 B includes the copy (com­puted). DeepSeek-V3 keeps its moments in BF16 [2], so its min­i­mum is 4 + 2 + 2 = 8 B per param­e­ter, or 10 B with the BF16 copy (com­puted). Table 12.2 gives the sizes. Spread over 16,384 GPUs, Llama 3 405B's 5.7 TB check­point is about 347 MB per GPU, or 396 MB at the 16 B upper bound of 6.5 TB (com­puted). Llama 3's stor­age sus­tains 2 TB/s with 7 TB/s peaks, and its authors name "the highly bursty check­point writes that sat­u­rate the stor­age fabric for short dura­tions" as a major chal­lenge [6].

PyTorch Dis­trib­uted Check­point writes at least one file per rank and reshards at load time, since check­points are often loaded on a new layout [925]. Load­ing is often the slow half, so MegaScale and Orbax read each shard once and broad­cast it [47, 926].

The design choice that mat­ters most is how long train­ing stops (Table 12.3). With an asyn­chro­nous check­point, the block­ing cost is only the copy to host memory, and the back­ground write sets a floor on the inter­val. Gemini's in-memory repli­cas raised the largest job's good­put from 85% to 97% [902], cut­ting lost time by a factor of 5 (com­puted).

How often: Young's inter­val, derived

Let $\delta$ be the block­ing cost of one check­point, $\tau$ the com­pute between check­points, $M$ the job's mean time to fail­ure (MTTF, Sec­tion 12.1) and $R$ the restart cost per fail­ure. Assume Pois­son fail­ures, inde­pen­dent of check­point times, and $\tau ,\delta ,R\ll M$. Check­points waste $\delta /\tau$. A fail­ure lands uni­formly inside an inter­val, so it loses $\tau /2$ of work plus $R$, at rate $1/M$:

Set­ting $d{f}_{\text{waste}}/d\tau =0$ gives Young's first-order inter­val [931], which Koko­lis et al. restate in node terms [903]:

ETTR, the effec­tive train­ing time ratio, is pro­duc­tive time over wall-clock time. At the opti­mum, check­point time equals expected rework. Dou­bling the job halves $M$, which short­ens ${\tau }^{*}$ by a factor $2^{1/2}\approx 1.41$ but dou­bles $R/M$ (com­puted): restart cost scales worse than check­point cost.

For an exact model, let fail­ures strike during check­points and restarts too, with a failed restart start­ing over. A stan­dard renewal argu­ment gives the expected wall time $T_w$ per unit of useful work $T_s$:

Min­i­mized numer­i­cally, its opti­mum is a little shorter than Young's: for a 300-second check­point and a 3.09-hour MTTF, about 2,390 s instead of 2,580 s (com­puted). Daly derived a higher-order closed-form esti­mate of this opti­mum [932].

Figure 65 turns this into a cal­cu­la­tor. Notice that the waste curve is flat near its floor, so a factor of two in $\tau$ costs little, and that once $\delta$ is a few sec­onds the restart band dom­i­nates what­ever inter­val you pick.

Koko­lis et al. give an expected ETTR for long jobs that wait little in the queue. Their $u_0$, $w_{cp}$ and $\Delta t_{cp}$ are our $R$, $\delta$ and $\tau$, and $n_{\text{nodes}}{r}_f=1/M$:

valid when $R+\tau /2\ll M$. They con­clude that a job using about 12,000 GPUs, two thirds of RSC-1, reaches an ETTR of 0.9 only if check­point writes take on the order of 10 sec­onds or the fail­ure rate falls from 6.50 to about 1 per 1,000 node-days [903]. Their for­mula with 5-minute restarts, at Young's inter­val, repro­duces this: 0.740 with 5-minute writes, 0.921 with 10-second writes and 0.898 at a fail­ure rate of 1 (com­puted).

BLOOM saved every three hours with 1⁠–⁠2 GPU fail­ures a week [401]. If that was Young-opti­mal at a 112-hour MTTF, $\delta ={\tau }^2/(2M)\approx 2.4$ min­utes (com­puted). Keep sev­eral check­points, since a diver­gence forces a roll­back: OPT restarted from points where its loss scale was still at least 1 [345]. And save on the SIGTERM that pre­cedes a pre­emp­tion [933].

The for­mula breaks when fail­ures cor­re­late with check­points, when the fail­ure rate drifts, or when $R+\tau /2$ is not small against $M$. To do it your­self, mea­sure block­ing and per­sist times sep­a­rately, com­pute ${\tau }^{*}$, round to whole steps no shorter than the per­sist time, and test restores with the over­lap test of Sec­tion 12.6.

With a 10-minute restart, a 5-second check­point stall and a 3-hour MTTF, is it worth more to halve the stall or the restart?

At the opti­mum, check­point waste is $(2\times 5/\mathrm{10,800}{)}^{1/2}=3.0\%$, and halv­ing the stall saves about 0.9 points. The restart term is $600/\mathrm{10,800}=5.6\%$, and halv­ing it saves 2.8 (com­puted). Fix the restart: waste grows as the square root of the stall but lin­early in $R$.

12.4 Getting back up: health checks, remediation and spares

A rank stops respond­ing at 3 a.m.: what brings the job back, how long does it take, and how do you keep bad nodes out of the next launch? An auto­mated loop brings it back: detect, local­ize, iso­late, recover. It starts before launch, because a big job is likely to draw a bad node. If a frac­tion $q$ of nodes is unhealthy and a job draws $n$,

which at $q=0.1\%$ is 1.6% for 16 nodes and 87% for Llama 3's 2,048 (com­puted). "One bad node spoils the bunch." [903]

The health-check ladder

In Imbue's bring-up of 4,088 H100s, about 10% of machines failed to boot, a quar­ter needed PCIe cables reseated, and after burn-in about 3% still break each week (reported) [935]. Table 12.4 lists the checks that follow. Meta tunes its peri­odic checks so that under 1% of suc­cess­ful jobs see a fail­ure [903].

These checks miss gray nodes, which pass every hard-fail­ure test but run slow. Guard com­pares nodes with their peers and reports step vari­ance cut from 20% to 1% [922]. SuperBench learns pass cri­te­ria from bench­mark dis­tri­bu­tions and reports up to 22.61x longer mean time between inci­dents [923]. Both are single-system reports. Exclu­sion has a cost too, since aggres­sive exclu­sion can starve a clus­ter of capac­ity [903].

The reme­di­a­tion loop

Detect. MegaScale watches RDMA traf­fic, which repeats every step, and starts recov­ery when it stops [47]. ByteRobust treats a 5x jump in loss or gra­di­ent norm, or a NaN, as a fault [904]. The slow­est detec­tor is the col­lec­tive time­out, the watch­dog that aborts a col­lec­tive that never com­pletes: pro­duc­tion runs have used 30 or 60 min­utes [937, 904] against PyTorch's cur­rent NCCL default of 10 min­utes [938].

Local­ize. Flight recorders find the rank that never joined (Sec­tion 12.6). Fail­ing that, group test­ing works: Imbue reran a fail­ing 48-machine job as six groups of eight, then eight groups of six [935]. Binary search over 2,048 nodes would take 11 launches (com­puted).

Iso­late. "ByteRobust favors rapid fault iso­la­tion over exhaus­tive local­iza­tion." [904] It evicts a whole sus­pect par­al­lel group rather than hunt for one GPU while thou­sands idle, but tol­er­ates two net­work alerts within 5 min­utes, since link flaps often clear [904].

Recover. MegaScale detects and diag­noses in under 10 min­utes and regains the pre-crash step within 15 [47]. With­out such a loop, Uni­cron's plain restart took about 68 min­utes [937]. Table 12.5 shows where restart time goes.

Hot spares

A hot spare is a val­i­dated idle node kept ready. BLOOM kept 4 for 48 work­ing nodes [401], OPT 18 for 124 [901] and Imbue 10⁠–⁠20% [935]. ByteRobust sizes its warm standby pool at the 99th per­centile of a bino­mial model of simul­ta­ne­ous fail­ures [904].

Beyond stop-the-world restart

The default launcher restarts every­one: "On fail­ures or mem­ber­ship changes ALL sur­viv­ing work­ers are killed imme­di­ately." [940] That works at Llama 3's scale.

torchft recov­ers a replica by copy­ing weights from a healthy peer, with no check­point load [941]. In RL post-train­ing, RobustRL restarts only the failed role (Sec­tion 9.8) [942]. As of Novem­ber 2026, public large dense runs still mostly evict and restart with spares. FT-HSDP's authors report no sig­nif­i­cant qual­ity impact, from exper­i­ments on 256 GPUs [939]. As of Novem­ber 2026 no public fron­tier run has con­firmed it.

12.5 What to watch: signals, alerts and logbooks

Which sig­nals tell you, within min­utes, that some­thing is wrong, and what kind of wrong? A crash is easy to notice. A drift is not, because a mis­con­fig­ured net­work usu­ally still trains: "Most of the time it will train but silently work a bit worse." [943] So log quan­ti­ties whose healthy shape you know (Table 12.6) and alert when the shape breaks. Why loss spikes happen is explained in Sec­tion 5.5. This sec­tion is about seeing them.

Loss, gra­di­ent norm and the rules that fire on them

Watch the gra­di­ent norm first. In OLMo 2 large gra­di­ent-norm spikes often pre­ceded loss spikes [7], and in GLM-130B col­lapse lagged a gra­di­ent-norm spike by a few steps [395]. The norm reacts to the batch at once, while Adam's per-coor­di­nate step stays near the learn­ing rate, so the weights need sev­eral steps to reach the region where the loss jumps.

OLMo 2's spike score is the per­cent­age of points at least seven stan­dard devi­a­tions from a rolling mean of the last 1,000 values. A change of ini­tial­iza­tion moved the gra­di­ent norm's score from 0.40 to 0.03 [7]. ByteRobust fires on a 5x jump [904]. The Deep Learn­ing Tuning Play­book logs the unclipped norm, clips at its 90th per­centile and calls clip­ping over half of updates aggres­sive [393]. Such a thresh­old clips 10% of a steady series by con­struc­tion, so a clipped frac­tion drift­ing toward 50% means the gra­di­ent scale moved (derived). Seven sigma is a rel­a­tive mea­sure. For Gauss­ian noise

so a Gauss­ian run would score 0, while heavy-tailed Stu­dent-t noise with 3 degrees of free­dom and no true spikes scores about 0.11% (com­puted). Com­pare scores between runs. No public study gives these rules' pre­ci­sion and recall.

Figure 66 lets you inject a fault and see which rule fires first. Look at the order: in a spike the gra­di­ent norm moves a few steps before the loss, a hang trips live­ness rules within min­utes while a 30-minute col­lec­tive time­out waits, and silent data cor­rup­tion (Sec­tion 12.2) leaves every panel flat until a replay com­par­i­son drifts. The BLOOM preset replays values from the real 176B train­ing log [945].

Inside the opti­mizer and the numer­ics

Adam's update per unit learn­ing rate lies between a pure-noise limit and a con­sis­tent-gra­di­ent limit (derived for ${\beta }_1=0.9$):

For Llama 3 405B at $\eta =8\times {10}^{-5}$ and an illus­tra­tive weight RMS of $1/128$, the update-to-weight ratio is $2.3\times {10}^{-3}$ to $1.0\times {10}^{-2}$ (com­puted). Watch for change, not the value. Moly­bog et al. tie diver­gence to large updates uncor­re­lated with descent [342], which here is a layer drift­ing toward the upper limit. Worts­man et al. found spikes 1⁠–⁠8 steps after squared gra­di­ents out­grow Adam's second moment [396]. After a jump by a factor $k$, the stal­e­ness is

which for $k=10$ stays above 2 for 6, 28 and 278 steps at ${\beta }_2=0.95$, 0.99 and 0.999 (com­puted).

For numer­ics, OPT saw diver­gence coin­cide with its loss scale crash­ing and final-layer acti­va­tion norms spik­ing [345]. In a mid-scale 53B-param­e­ter MoE trained with plain Muon, the Kimi team saw max­i­mum atten­tion logits exceed 1,000. Kimi K2, trained with logits clipped at 100 (QK-Clip), reports 15.5T tokens with­out a loss spike [72]. A kernel that skips the row-max­i­mum sub­trac­tion over­flows $e^x$ above 11.09 in FP16 and 88.72 in BF16 (com­puted).

Per­for­mance, hard­ware and live­ness

Know your MFU for­mula: llm.c's 49.0% for GPT-2 counts atten­tion FLOPs and is 42.5% with 6N alone (com­puted from [15], con­ven­tions in Sec­tion 1.3). Hard­ware speaks through Xid events (Table 12.7), and ECC errors lay behind 38.9% of faults in Minder's data [946].

Log­ging has costs. Print­ing a GPU scalar forces a syn­chro­niza­tion, so OLMo 2 logs every 10 or 50 steps and does book­keep­ing on a sep­a­rate Gloo group [7]. Mon­i­tors fail too: at Acme many con­nec­tion errors came from aux­il­iary metric and log­ging ser­vices [905]. And no single signal suf­fices [946, 934], so alert on sev­eral, in tiers (our syn­the­sis of OPT, MegaScale and ByteRobust prac­tice): page at once for no progress (a stale log, zero RDMA traf­fic, a miss­ing heart­beat), a NaN loss or a crash [901, 47, 904], page in work­ing hours for a spike, a sus­tained through­put drop or a loss scale pinned low, and file a ticket for slow drifts such as gra­di­ent-norm growth or rising cor­rectable ECC on one node.

12.6 Debugging playbooks: spikes, NaNs, resumes, hangs and slowdowns

The loss spiked, a NaN appeared, a resumed run drifted, or the job hung: what do you do, and in what order? Save the evi­dence before you touch any­thing, find the small­est thing that repro­duces the prob­lem, and change one thing at a time.

Loss spikes

Acme counts a spike only if a fall­ing loss jumps and "does not recover over a cer­tain period" [905]. For one that does not recover, read the lead­ing sig­nals (Sec­tion 12.5), keep the check­points on either side and inspect the batches: OLMo 2 found long repeated n-grams often behind spikes, "though we note that this rela­tion­ship is not deter­min­is­tic" [7]. Then rewind to a healthy check­point and skip the next batches [905]. PaLM's restart-and-skip fix is told in Sec­tion 5.5. If spikes recur, change the recipe, low­er­ing the learn­ing rate last [393].

The cause is con­tested: data, data com­bined with the param­e­ter state, Adam's dynam­ics [342, 396], grow­ing atten­tion logits [278] or silent data cor­rup­tion [914]. The views fit together: data trig­gers, opti­mizer and archi­tec­ture set the sen­si­tiv­ity, hard­ware is a wild­card. Recent fron­tier runs self-report few spikes or none [6, 2, 72], so as of Novem­ber 2026 spikes look like a recipe prob­lem more than a scale prob­lem.

NaNs, drift­ing resumes and parity

Table 12.8 covers the other numer­i­cal symp­toms.

The test for a clean resume is a delib­er­ate over­lap. PaLM guar­an­tees that a restart from check­point 15,000 repro­duces steps up to 17,000 exactly [34], and ByteRobust rolls back a few steps after manual restarts to check bit­wise align­ment [904]. With­out bit­wise deter­min­ism, com­pare against noise mea­sured once (Sec­tion 12.7). Rerun a small FP32 parity test on every frame­work or kernel upgrade.

Hangs

A hung job logs noth­ing until a col­lec­tive time­out fires (Sec­tion 12.4), and implicit hangs were 9.9% of inci­dents across ByteRobust's plat­form [904].

Figure 67 walks a 32-rank toy job through one stalled GPU. Watch 31 ranks log time­outs while the faulty one stays silent, then find it by its lag­ging sequence number and odd stack.

The flight recorder keeps, per rank and process group, the last col­lec­tives enqueued, started and com­pleted [950]. The cul­prit sits at "the first col­lec­tive where some ranks started the col­lec­tive but others did not" [903]. At 16,384 ranks and 2,000 entries each that is 32.8 mil­lion entries (com­puted), so tool­ing must rank sus­pects.

Your 16,384-GPU job hung and rank 412 logged a col­lec­tive time­out. Is rank 412 the cul­prit?

Prob­a­bly not: wait­ing ranks time out and log, while the stuck rank often logs noth­ing [47]. Dump the flight recorder on every rank (dump­ing on time­out is off by default), find the rank that never joined or the odd stack, and evict its par­al­lel group.

Time­outs are a knob, not a fix. NCCL's InfiniBand wait is

for time­out expo­nent $x$ (NCCL_IB_TIMEOUT) and retry count $c$. The defaults $x=20$ and $c=7$ give 4.29 s per try and 30.1 s total, the doc­u­mented "approx­i­mately 30 sec­onds of wait­ing" [952], and $x=23$ gives 4.0 min­utes (com­puted). MegaScale raised time­outs to sur­vive NIC flaps [47], but a real hang then waits longer. Pair a mod­er­ate time­out with a watch­dog on log fresh­ness and traf­fic.

Slow­downs

Check the per-rank step-time spread first. A wide spread means a strag­gler, a narrow one the pro­gram, and garbage col­lec­tion is a common cul­prit [935, 919].

If unsure, com­pare MFU with the arith­metic (Sec­tion 1.3), then work down the table.

12.7 Determinism and reproducibility

Same code, same seed, same data: why do two runs differ, and when is it worth paying to make them iden­ti­cal? Float­ing-point addi­tion is not asso­cia­tive. In double pre­ci­sion

(com­puted), because 0.1 is lost when added to ${10}^{20}$ first. Sum­ming eight num­bers in 10,000 random orders, Think­ing Machines found "there are 102 unique results" [848]. Rerun­ning the pub­lished script on Python 3.11 gives 258, and all 40,320 orders give 275 dis­tinct sums (com­puted). The exact count varies with the setup. The point does not. Any change in reduc­tion order can change the bits. Table 12.11 lists the sources in the order you remove them.

Why pay? BF16 greedy decod­ing can vary by up to 9% in accu­racy across GPU counts, types and batch sizes [402], and in RL a sam­pler and trainer that dis­agree make on-policy train­ing partly off-policy (Sec­tion 9.6) [683]. In train­ing, deter­min­ism speeds root-caus­ing, hard­ware faults included [34, 902, 914]. Remov­ing one source does not remove run-to-run vari­ance, since one-bit changes send models to dif­fer­ent solu­tions [961]. A prac­ti­cal default: deter­min­is­tic set­tings for debug­ging, parity tests and eval­u­a­tion, fast ker­nels in pro­duc­tion unless you need bit­wise replay, and one mea­sure­ment of run-to-run noise.

Why does a tem­per­a­ture-0 API some­times give dif­fer­ent answers to the same prompt?

Not atomic adds. Ker­nels choose their reduc­tion strat­egy by batch size, and batch size fol­lows server load, so your request is summed in an order set by its neigh­bors. Batch-invari­ant ker­nels made 1,000 of 1,000 com­ple­tions iden­ti­cal, at 2.1x the time in an unop­ti­mized demo and 1.6x with a better atten­tion kernel [848] (com­puted).

12.8 Goodput and scheduling: accounting for every GPU-hour

Of every 100 GPU-hours you pay for, how many turn into train­ing progress, and who owns each loss? Google splits the answer three ways [933]. Sched­ul­ing good­put (SG) is the frac­tion of time all the job's resources are avail­able, run­time good­put (RG) the frac­tion of that time making for­ward progress, and pro­gram good­put (PG) the model FLOPs uti­liza­tion. Each is a frac­tion of the one before, so on our read­ing they mul­ti­ply into ML Pro­duc­tiv­ity Good­put (MPG):

Sched­ul­ing loses to queue­ing and pre­emp­tion, run­time to restarts, stalls and lost work, pro­gram good­put to exposed com­mu­ni­ca­tion and slow ker­nels. MFU and its con­ven­tions (6N or with atten­tion, which peak, which time base) are defined in Sec­tion 1.3. Say which one a number uses. Since fail­ures and the cost of idle hours both grow with clus­ter size, "the over­all cost of fail­ure rises quadrat­i­cally with the scale of the com­pute clus­ter" [933].

MPG is 0.36 with run­time good­put 0.90 and sched­ul­ing good­put 0.98. What is the MFU, and which buys more: run­time good­put of 0.97 or 5 more MFU points?

$0.36/(0.98\times 0.90)=0.41$. Run­time good­put of 0.97 gains $0.97/0.90=1.078$, five MFU points $0.46/0.41=1.12$ (com­puted). MFU wins, which is why each factor needs an owner.

Sched­ul­ing: all the GPUs at once

A syn­chro­nous job needs every rank at step 1, so the sched­uler starts all of it or none, which Vol­cano calls gang sched­ul­ing [963]. Slurm uses the same term for times­lic­ing [964]. Place­ment mat­ters as much as admis­sion: on an unop­ti­mized large clus­ter Meta saw com­mu­ni­ca­tion uti­liza­tion "rang­ing from 10% to 90%" until it com­bined topol­ogy-aware sched­ul­ing with rout­ing and NCCL changes [145]. Small jobs starve: Acme's eval­u­a­tion jobs queued long­est [905]. By 2026 Slurm versus Kuber­netes is a tool­ing choice, and the largest oper­a­tors write their own [965, 966].

12.9 Lineage, data pipelines, evaluation gates and technical debt

What keeps a months-long run repro­ducible, fed and judged, and why does that machin­ery rot? A fron­tier run is a sequence of seg­ments, each with its own code, con­fig­u­ra­tion and data. Llama 3 changed its data mix sev­eral times during train­ing [6], and on the ByteRobust plat­form 9,582 manual restarts for code or data adjust­ments made up 17.3% of three months' inci­dents [904].

Record each seg­ment as in Table 12.14 and follow Imbue's rule to "change only one thing at a time" [935]. Oth­er­wise a code change and a hard­ware swap land together, and nobody can say which moved the loss.

Data pipe­lines

Pipe­lines are built for exact resump­tion, not band­width. Sequen­tial tar shards read "3x-10x" faster than random access [975]. StreamingDataset keeps sample order inde­pen­dent of GPU count [976], and StatefulDataLoader check­points worker state but not across ranks, so each rank saves its own [977]. Make the global order a pure func­tion of seed and sample index, as PaLM's step-indexed pipe­line does [34]. Then resum­ing and rewind-and-skip are arith­metic.

Eval­u­a­tion pipe­lines and pro­mo­tion gates

On Acme's Kalos clus­ter, eval­u­a­tion was 92.9% of jobs but 0.8% of GPU time [905]. Scores are noisy: 1B models span 1.7 ARC Chal­lenge points over their last 30 check­points [259]. Log per-exam­ple like­li­hoods [258], smooth over check­points before alert­ing, pin the har­ness [497, 775] and keep a held-out suite [7] (method­ol­ogy in Sec­tion 10.9). A pro­mo­tion gate is a rule over tracked eval­u­a­tions that moves a reg­istry alias [978]. Llama 3 advanced long-con­text stages only after short-con­text scores recov­ered and needle-in-a-haystack tasks were solved [6] (Chap­ter 7).

Why the machin­ery rots

Scul­ley et al.'s risk fac­tors for ML sys­tems [979] map onto a pre­train­ing team as in Table 12.15 (our syn­the­sis). The shared defense is lin­eage plus tests for oper­a­tions code, which the ML Test Score grades [980].

13. Inference: serving models fast and cheap

Serv­ing pays repeat­edly for the model that train­ing pro­duced. Pre­fill and decode have dif­fer­ent bot­tle­necks, and batch­ing trades latency against uti­liza­tion. Count the weight reads, KV memory, queue­ing and accepted output tokens before attach­ing a price to a request.

On 27 and 28 Feb­ru­ary 2025, DeepSeek's V3 and R1 ser­vice ran on an aver­age of 226.75 nodes of 8 H800s, about 1,814 GPUs (reported, GPU count com­puted) [14], close to the clus­ter of 2,048 H800 GPUs that pre­trained DeepSeek-V3 [2]. That day it read 608B input tokens and wrote 168B output tokens [14]. Serv­ing, not train­ing, is where a suc­cess­ful model spends its life.

A request waits in a queue, then the server runs the whole prompt in one pass, called pre­fill, ending with the first output token. That delay is the time to first token (TTFT), and it includes queue­ing, pre­fill and net­work time [981]. Then decode runs one for­ward pass per new token until the answer stops. The gap between tokens is the time per output token (TPOT) [981], defined exactly in Sec­tion 13.3.

The two phases run at very dif­fer­ent speeds on the same chips. On 64 TPU v4 chips, PaLM 540B pre­filled 2,048 tokens in 0.29 s and gen­er­ated 64 tokens at batch 64 in 1.82 s [33]: 0.14 ms per prompt token against 28.4 ms per decode step, so one sequence waits about 200 times longer per gen­er­ated token. Each step served 64 sequences, so the chip time per gen­er­ated token was 0.44 ms, about 3 times a prompt token (com­puted).

The anchor for every number is Llama 3 70B in BF16 on one node of 8 H100s [6], with DeepSeek-V3 for mix­ture of experts and GPT-2 124M on a laptop. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3. Sec­tions 13.1 to 13.3 do the arith­metic, 13.4 to 13.6 build the serv­ing system and 13.7 to 13.9 make decode cheaper, build­ing on Chap­ter 1 (2N and the roofline), Chap­ter 2 (band­widths) and Chap­ter 4 (atten­tion vari­ants and experts).

13.1 Prefill and decode: two workloads in one request

Why does a server read a 4,000-token prompt faster than it writes 100 tokens of answer? Pre­fill sees every prompt token at once, so each weight read from memory serves thou­sands of tokens. Decode cannot start token $t+2$ before fin­ish­ing token $t+1$, so every step reads every weight to make one token per sequence. Split­wise's authors call these "a com­pute-inten­sive prompt com­pu­ta­tion phase and a memory-inten­sive token gen­er­a­tion phase" [92].

Pre­fill runs the $s_p$ prompt tokens through the model, keeps every layer's keys and values (the KV cache) and sam­ples the first token [33]. Each decode step runs one new token per sequence, attends to the cache, appends its own keys and values and sam­ples. With­out the cache, in kipply's words, "sam­pling would be qua­dratic in time com­plex­ity as we increase the number of tokens" [982].

FLOPs and bytes

The weight mat­muls cost about $2N$ FLOPs per token, one mul­ti­ply and one add per param­e­ter [33]. Causal atten­tion in pre­fill costs about $2s_p^{2}h{d}_{h}L$ FLOPs, which equals the weight FLOPs at $s_{\text{eq}}=N/(h{d}_{h}L)=70.55\times {10}^9/(\mathrm{8,192}\times 80)\approx \mathrm{107,700}$ tokens for Llama 3 70B [6]. Atten­tion adds 8% to the weight FLOPs at 8K, 30% at 32K and 122% at 128K (com­puted). So "2N per token" holds below about 32K tokens and fails at 128K.

Speed is set by arith­metic inten­sity, FLOPs per byte moved. The roofline and the ridge point are derived in Sec­tion 1.7. Apply them here with­out re-deriv­ing. For BF16 weights,

where $F$ is a GPU's peak FLOP/s and $\text{BW}$ its HBM band­width. Divid­ing the H100 SXM's dense BF16 peak, 989.5 TFLOP/s (half the 1,979 TFLOP/s the spec sheet lists with spar­sity, see Sec­tion 2.1), by its 3.35 TB/s of HBM band­width puts the ridge at about 295 FLOP per byte (com­puted) [4]. Zadouri et al. use the same value of about 295 [983], while Austin et al. put the crit­i­cal batch for an H100 at "about 280 tokens" [91]. Where the dif­fer­ence does not matter, this chap­ter says about 300. Quan­tiz­ing weights to 8 bits halves $b_{\text{crit}}$, and 8-bit math dou­bles it [91].

Atten­tion breaks the batch­ing argu­ment

Batch­ing fixes the weight mat­muls, since one weight read serves the whole batch, but not atten­tion, since each sequence has its own KV cache. In decode, multi-head atten­tion does about one FLOP per KV byte, grouped-query atten­tion about $h/h_{kv}$ and latent atten­tion about $2h$ [983]: 8 for Llama 3 70B, about 256 for DeepSeek-V3. In gen­er­a­tion, Austin et al. con­clude, "we are always memory band­width-bound" [91], which is why Llama 3 uses grouped-query atten­tion "with 8 key-value heads to improve infer­ence speed and to reduce the size of key-value caches during decod­ing" [6]. Llama 3 70B's cache is 327,680 B per token, against 70,272 B for DeepSeek-V3's latent atten­tion (Sec­tion 4.5).

Now write one decode step for $b$ sequences at con­text $s$, with $W$ bytes of weights, $m_{\text{kv}}$ KV bytes per token and $p_{\text{tp}}$ tensor-par­al­lel GPUs:

Each added sequence brings $2N$ FLOPs and $s{m}_{\text{kv}}$ bytes, and above $s^{*}$ its bytes take longer to read than its FLOPs take to com­pute. For Llama 3 70B in BF16 on 8 H100s, the batch that reaches the ridge is 295 sequences at zero con­text, 455 at 512 tokens and 992 at 1,024 tokens, and above about 1,460 tokens of con­text no batch size makes the step com­pute-bound (com­puted at 100% of peak band­width and FLOP/s, ignor­ing atten­tion FLOPs, which would move the thresh­old to about 1,500 tokens) [6, 4]. Pro­fil­ing agrees: Recasens et al. find large-batch decode memory-bound, "with over 50% of the atten­tion kernel cycles stalled due to data access delays for all tested models" [984]. Both hold: Austin et al.'s crit­i­cal batch con­cerns the weight mat­muls, this thresh­old the whole step.

Step through Figure 6 with its Next button to plot this request: pre­fill, then decode at batch 1, then at batch 64. Batch 64 moves decode right but nowhere near the ridge.

So report input and output tokens sep­a­rately, since a gen­er­ated token costs sev­eral times as much as a prompt token even at large batch (Sec­tion 13.3), and check that research code uses a KV cache at all.

13.2 The decode floor: bytes over bandwidth

What is the fastest a model can gen­er­ate one token, and how many users fit on one node? A decode step cannot end before every weight byte and every cached KV byte has crossed from HBM to the com­pute units, the memory roof of Sec­tion 1.7. Austin et al. call this the "The­o­ret­i­cal Min Step Time" [91], Pope et al. memory time [33]:

It assumes per­fect over­lap, peak band­width, free com­mu­ni­ca­tion and neg­li­gi­ble acti­va­tions. For Llama 3 70B in BF16 on one node of 8 H100s with tensor par­al­lel­ism of 8, every decode step streams 141.1 GB of weights over 8 × 3.35 TB/s of HBM band­width, so its floor is 5.27 ms per token at batch 1, about 190 tokens/s for one user, and 5.37 ms at 8K tokens of con­text (com­puted) [6, 4, 91]. Table 13.1 applies the bound to the other models [84].

The last row shows that pipe­line par­al­lel­ism adds capac­ity, not speed. A token visits the stages in sequence, so the floor divides by $p_{\text{tp}}$, not by the GPU count. Micro-batches in flight restore through­put, which is how Meta served 405B in BF16 "across 16 GPUs on two machines" [6].

KV memory sets con­cur­rency

Llama 3 70B's BF16 cache takes $2\times 80\times 8\times 128\times 2=\mathrm{327,680}$ bytes (320 KiB) per token [6], 2.68 GB per 8K-token sequence. BF16 weights leave $640-141=499$ GB on 8 H100s, room for 185 such sequences if every free byte held KV. Real­is­tic head­room shrinks the pool to about 150 sequences (Sec­tion 13.4), so say "about 150 to 185" and state the head­room. At long con­text the cache out­grows the model: Llama 3 8B's KV at 128K tokens is 17.2 GB, its BF16 weights 16.1 GB (com­puted). KV size moves through­put too. In Austin et al.'s LLaMA 2⁠–⁠13B exam­ple on 8 TPU v5e chips, a 5 times smaller cache raises the­o­ret­i­cal through­put at batch 240 from 964 to 4,529 tokens/s, although nei­ther batch fits in the chips' 128 GiB [91].

The latency-through­put fron­tier

Table 13.2 applies the step for­mula to the anchor at 4K tokens of con­text [6, 4].

From batch 1 to 256, total through­put rises 75 times while each user's speed falls 3.4 times. Going from 64 to 128 buys only 1.45 times more, because KV traf­fic grows with the batch, and even batch 256 stays memory-bound (com­puted). On one A100, Databricks found that "if we max­i­mize through­put with a batch size of 64, latency increases by 4x while through­put increases by 14x" [985]. CloudMatrix-Infer, serv­ing DeepSeek-R1 in INT8 on Ascend 910 NPUs, reports 1,943, 974 and 538 tokens/s per NPU at TPOT tar­gets of 50, 30 and 15 ms [986]. Figure 68 turns the fron­tier into a cal­cu­la­tor. Try FP8 weights (the batch-64 step falls to 5.84 ms) and a 32K con­text (KV traf­fic flat­tens the curve).

From floor to real­ity

Databricks scores deploy­ments by model band­width uti­liza­tion, $\text{MBU}=((W+s{m}_{\text{kv}})/\text{TPOT})/\text{peak bandwidth}$ at batch 1, and mea­sured 60% for a 7B model on 2 H100s [985]. Applied to the anchor as an illus­tra­tive assump­tion, 60% turns 5.27 ms into 8.8 ms, about 114 tokens/s. So expect roughly 100 to 130 tokens/s at batch 1, a com­puted esti­mate, since we found no pub­lished mea­sure­ment of this setup. Tensor par­al­lel­ism adds latency too: at batch 1 each all-reduce costs a fixed delay, which kipply guesses at "approx­i­mately 8 microsec­onds per mes­sage" from a V100 NVLink mea­sure­ment [982]. Mega­tron-style tensor par­al­lel­ism does two all-reduces per layer in the for­ward pass [392], 160 over 80 layers, or about 1.3 ms, a quar­ter of the floor (com­puted, assump­tion-sen­si­tive). That is why, at small batch, "for Llama2⁠–⁠70B going from 4x to 8x GPUs only decreases latency by 0.7x" [985].

Do it your­self. Apple lists 153 GB/s of memory band­width for the MacBook Air (as of Novem­ber 2026) [987]. GPT-2 124M's FP32 weights are $\mathrm{124,439,808}\times 4$ B = 498 MB, a floor of 3.25 ms per token (about 308 tokens/s), and 1.63 ms in BF16. Its KV is $2\times 12\times 12\times 64\times 2=\mathrm{36,864}$ B per token in BF16 (com­puted) [19]. Mea­sure tokens/s and com­pute your MBU: Python and kernel launches dom­i­nate at this size.

Three mis­takes recur. Leav­ing train­ing-style weight shard­ing on during decode is "an easy and common way to have order of mag­ni­tude regres­sions" [91]. Sizing the batch from weights alone runs out of memory when long prompts arrive. And a floor is a bound, never a mea­sure­ment.

13.3 Latency metrics, goodput and the price of a token

What should you mea­sure, and what does a mil­lion tokens cost to pro­duce? TTFT runs from sub­mis­sion to the first token and "gen­er­ally includes request queu­ing time, pre­fill time, and net­work latency" [981]. NVIDIA's AIPerf defines inter-token latency, which this chap­ter calls TPOT, over the $n_{\text{out}}$ output tokens [981]:

"A TPOT of 100 mil­lisec­onds/tok would be 10 tokens per second per user, or ~450 words per minute" [985]. Good­put is "the max­i­mum request rate that can be served adher­ing to the SLO attain­ment goal (say, 90%) for each GPU pro­vi­sioned" [988], where an SLO (ser­vice-level objec­tive) is a latency target.

Def­i­ni­tions dis­agree: "tools differ on whether TTFT is included in the aver­age" [981], and Databricks' latency, TTFT plus TPOT times the output length, counts one more gap [985]. Tok­eniz­ers differ too: "Llama 2 tok­eniza­tion is 19% longer than ChatGPT tok­eniza­tion" in Anyscale's mea­sure­ment [985]. Report TTFT and TPOT as p50, p90 and p99 at a stated request rate and length mix, plus good­put at a stated SLO.

Queue­ing inflates TTFT before through­put sat­u­rates. DistServe models a pre­fill instance as "the M/D/1 queue", with Pois­son arrivals at rate $R$ and fixed pre­fill time $D$ [988]:

At 80% uti­liza­tion ($RD=0.8$) the queue­ing term is $2D$, so aver­age TTFT is three times the bare pre­fill (com­puted). A TTFT target forces head­room.

Users feel some­thing else. Etalon's authors say these met­rics "fail to fully cap­ture the nuances of LLM infer­ence", and a 250-words-per-minute reader needs only about 6 tokens/s [989]. Andes adds that "gen­er­at­ing tokens at a rate faster than the user's con­sump­tion speed should not improve QoE" (qual­ity of expe­ri­ence) [990]. Fast decode mat­ters most for agents and rea­son­ing models that read their own output (Chap­ter 9, Chap­ter 14).

The price of a token

For $p$ GPUs at $c$ dol­lars per GPU-hour sus­tain­ing $X$ tokens per second,

Pope et al.'s chip-sec­onds per token is the same quan­tity with­out a price [33]. The $2 per GPU-hour used here is DeepSeek's own assump­tion [14], reused for DeepSeek-V3.2's esti­mates [681], not a market quote. State uti­liza­tion, length mix, the batch the latency target allows and the price.

Decode runs at low MFU, holds KV memory for the whole gen­er­a­tion and has its batch capped by the TPOT target. Step-3's authors call decod­ing "the most expen­sive per token (because of low MFU) com­pared with train­ing and pre­fill" [991]. Market prices show the same ratio. In Feb­ru­ary 2025, DeepSeek-R1 charged $0.14 per mil­lion input tokens on a cache hit, $0.55 on a miss and $2.19 per mil­lion output tokens [14], 4.0 times for output (com­puted). As of Novem­ber 2026, DeepSeek's price page lists deepseek-v4-pro at $0.044, $1.32 and $3.96 at peak hours [992]: 3.0 times for output, and a cache hit 30 times cheaper than a miss (com­puted), because a hit skips pre­fill (Sec­tion 13.4). The price con­trol of Figure 68 puts dol­lars on any con­fig­u­ra­tion.

Serv­ing cost feeds back into train­ing. Sar­dana et al. esti­mate that for 2 tril­lion tokens of infer­ence demand, train­ing a 7B model on more data instead of a 13B model cuts total com­pute by 17% [70]. Sizing a model for its serv­ing load is Sec­tion 6.5.

Cost esti­mates fail in three ways. They assume full load around the clock, yet DeepSeek serves "across all nodes during peak day­time hours" and at night moves nodes to research and train­ing [14]. They forget that rea­son­ing models write long out­puts, so output cost dom­i­nates. And they com­pare per-token prices across dif­fer­ent tok­eniz­ers.

13.4 Batching and KV memory: continuous batching, paging and prefix caching

How does one GPU keep hun­dreds of requests in flight when they arrive and finish at random times? A serv­ing engine is a batch­ing machine under a memory con­straint. It wants many tokens in every for­ward pass, since each pass streams the weights once (Sec­tion 13.2), but every request in flight holds a grow­ing KV cache. Pass time depends on "the raw number of tokens in the for­ward pass", not on how many sequences they come from [993]. So the sched­uler's job is to keep passes full with­out run­ning out of KV memory.

Static batch­ing wastes slots

A clas­sic server runs a batch until every request in it fin­ishes. A slot whose request ends early idles until the long­est one ends. With inde­pen­dent output lengths $\ell$, the share of slot-steps doing useful work is

and for lengths uni­form on $1,\dots ,M$

With output lengths uni­form on 1 to 128 tokens, the range Orca's eval­u­a­tion used [994], a static batch keeps 56% of its slots busy at batch 8 and 52% at batch 32 (com­puted and checked by sim­u­la­tion). For an illus­tra­tive heavy-tailed work­load (expo­nen­tial, mean 200) it is 37% and 25%. Bigger batches make it worse.

Orca's fix is iter­a­tion-level sched­ul­ing, which "sched­ules exe­cu­tion at the gran­u­lar­ity of iter­a­tion (instead of request)" [994]. A fin­ished request leaves at once and a queued one takes its slot at the next step. This is con­tin­u­ous batch­ing (in-flight batch­ing in TensorRT-LLM [995]). At sat­u­ra­tion it raises decode through­put by about $1/U$: 1.9x at batch 32 for the uni­form work­load, 4.1x for the heavy-tailed one (com­puted). Linear layers batch every request's tokens while atten­tion runs per request (selec­tive batch­ing) [994].

Paging the KV cache

Orca still reserved each request's max­i­mum length up front. Kwon et al. mea­sured that exist­ing sys­tems stored real tokens in only 20.4% to 38.2% of their KV memory. The rest went to slots reserved for future tokens, over-pro­vi­sion­ing for lengths that never came and frag­men­ta­tion [73]. PagedAttention bor­rows paging from oper­at­ing sys­tems:

1. Cut the KV pool into fixed blocks, 16 tokens by default in vLLM [73].
2. Give each request a block table that maps its log­i­cal blocks to phys­i­cal blocks any­where in the pool. Sam­ples of one prompt can share blocks and copy one only when they write to it.
3. Allo­cate a block only when the last one fills, so waste stays under one block per request [73]: under 5.2 MB for Llama 3 70B, 2.6 MB on aver­age, against 2.36 GB for an 8,192-token reser­va­tion that ends at 1,000 tokens (com­puted).
4. Gather keys and values through the table inside the atten­tion kernel, and return blocks to a free list when the request ends.

When blocks run out, the engine pre­empts a request and later recom­putes its KV, the vLLM V1 default [996]. vLLM raised through­put 2⁠–⁠4x at equal latency over FasterTransformer and Orca [73] (mea­sured, 2023 base­lines). Its paged kernel was slower, up to 2.8x against FlashAttention-2 in vAttention's mea­sure­ment. vAttention instead keeps the cache vir­tu­ally con­tigu­ous and allo­cates phys­i­cal memory at run­time, which raised serv­ing through­put by up to 1.23x over paged FlashAttention-2 and FlashInfer ker­nels [997] (mea­sured). Both designs avoid reserv­ing max­i­mum lengths.

Figure 69 runs one illus­tra­tive work­load under three poli­cies. Step it and watch the gray idle slots vanish under con­tin­u­ous batch­ing, then the hatched reserved-but-empty blocks vanish under paging. Shrink the KV pool to see memory cap the batch.

Prefix caching

Under causal atten­tion a token's keys and values depend only on ear­lier tokens, so requests that share a prefix can share its blocks. vLLM keys each full block by its parent's hash plus its own tokens [999], at under 1% through­put cost even with no hits [1000]. SGLang's RadixAttention keeps a radix tree of cached tokens with least-recently-used evic­tion and serves the long­est shared prefix first, which its authors warn "can lead to star­va­tion" [1001]. Real hit rates are high: 74.1% for Vicuna-33B on Chat­bot Arena [1001] and 56.3% of DeepSeek's 608B daily input tokens in Feb­ru­ary 2025 [14]. A 4,000-token shared prefix with a 200-token ques­tion cuts linear-layer pre­fill work from $2N\times \mathrm{4,200}$ to $2N\times 200$ FLOPs, 21x (com­puted). Atten­tion to the cached posi­tions still runs.

Batch­ing also serves many LoRA adapters (Sec­tion 8.3) on one base model: com­put­ing $xW+xAB$ unmerged lets $xW$ batch across ten­ants, and S-LoRA serves 2,000 adapters at once [1003] (reported). A rank-16 adapter on Llama 3 8B's atten­tion pro­jec­tions is 0.17% of the model (com­puted) [6].

13.5 Prefill meets decode: chunked prefill and disaggregation

When a long prompt arrives, what hap­pens to the users already mid-answer, and should pre­fill and decode share GPUs at all? If the engine admits the whole prompt at once, every decod­ing user waits for its pre­fill. Pri­or­i­tiz­ing pre­fills favors through­put and hurts the tail of time between tokens (TBT, the indi­vid­ual gaps whose mean is TPOT, Sec­tion 13.3), and pri­or­i­tiz­ing decodes does the oppo­site [1004]. On the anchor node an 8,192-token pre­fill at an illus­tra­tive 40% MFU takes about 393 ms, while a decode step for 32 users at 4K con­text takes 6.9 ms (com­puted). One admis­sion makes every user's next gap 57 times longer.

Chun­ked pre­fill

Sarathi splits the prompt into chunks and fills the rest of each batch with decodes, which ride almost free because the chunk already pays for stream­ing the weights. On LLaMA-13B, a decode token cost 12.49 ms alone and 1.2 ms pig­gy­backed on a chunk [1005] (mea­sured). Sarathi-Serve gives each iter­a­tion a token budget, filled first with every ongo­ing decode and then with the next prompt chunk. The budget is the largest that keeps an iter­a­tion inside the TBT target: 2,048 tokens for relaxed and 512 for strict SLOs in the paper (1,536 for Llama 2 70B, relaxed) [1004]. A few hun­dred tokens suf­fice because a pass turns com­pute-bound near the ridge, 295 FLOP per byte on an H100 (Sec­tion 13.1). Against vLLM of early 2024, capac­ity under a P99 TBT target rose 2.6x for Mis­tral-7B [1004] (mea­sured). vLLM V1 now chunks by default and sched­ules decodes first, and its guide says smaller bud­gets favor inter-token latency and larger ones the time to first token [996]. A prompt of $s_p$ tokens needs

iter­a­tions when $n_{\text{dec}}$ decodes ride along.

Dis­ag­gre­ga­tion

Chunk­ing bounds the stall. Dis­ag­gre­ga­tion removes it: pre­fill runs on one instance, which com­putes the prompt's KV and the first token, then ships the KV to a decode instance that adds the request to its batch. Each phase picks its own par­al­lel­ism and batch size, and each pool scales on its own. DistServe built it to max­i­mize per-GPU good­put (Sec­tion 13.3) and served 7.4x more requests or met 12.6x tighter SLOs than colo­cated sys­tems [988] (mea­sured).

The price is the KV trans­fer, a nearly fixed frac­tion of pre­fill time. Bytes moved and pre­fill FLOPs both grow lin­early with the prompt length $s_p$, so it can­cels:

where ${\text{BW}}_{\text{link}}$ is the link band­width per direc­tion and atten­tion, a small share of pre­fill, is ignored. Mea­sured sys­tems hide most of it. In DistServe the trans­fer was under 0.1% of total latency for OPT-175B [988]. Split­wise's layer-by-layer trans­fer added 0.8% to end-to-end latency but 16.5% to the second token [92]. Those shares divide by end-to-end latency, which includes every decode step, so they do not check the ratio above.

Figure 70 admits one long prompt into a node already decod­ing for 32 users. Watch the red worst-gap marker: it spans the whole pre­fill when colo­cated, one chunk when chun­ked and one decode step when dis­ag­gre­gated. Then change the link and see the green trans­fer block shrink.

Split phases can run on dif­fer­ent hard­ware: Split­wise found that token gen­er­a­tion barely slows under a power cap from 700 to 350 W, while the prompt phase is highly sen­si­tive [92]. Moon­cake, which serves Kimi's pre­fill-heavy traf­fic (8,590 input and 182 output tokens per request on aver­age), adds a clus­ter-wide KV cache in CPU memory and SSD and lets Kimi handle 75% more requests [1006] (reported).

Whether to split is still debated. DistServe's authors argue that one large pre­fill can inflate time per output token 2 to 30 times under bursty load even with chunk­ing [1007]. TaiChi finds colo­ca­tion best for tight first-token tar­gets and dis­ag­gre­ga­tion best for tight per-token tar­gets [1008]. Large deploy­ments now do both: chunk­ing inside pools, dis­ag­gre­ga­tion across them.

13.6 Serving a mixture of experts: DeepSeek-V3 at scale

DeepSeek-V3 acti­vates 37B of its 671B param­e­ters per token. Is it as cheap to serve as a 37B dense model? In FLOPs, nearly: about $2\times 37\times {10}^9=74$ GFLOPs of weight math per token (com­puted) [2]. In memory, no. All 671 GB of FP8 weights must sit in HBM, and each expert sees only a slice of the batch, so its weights are read for little work. Sparse acti­va­tion "shifts feed-for­ward net­works (FFNs) from being com­pute-inten­sive to memory-inten­sive during infer­ence" [1014]. The archi­tec­ture is in Sec­tion 4.7.

With $E$ routed experts and $k$ active per token, a bal­anced step of $b$ tokens gives each expert $bk/E$. For every expert to reach the ridge,

DeepSeek-V3 acti­vates 8 of 256 routed experts [2], so a step needs about $295\times 256/8\approx \mathrm{9,450}$ tokens (com­puted, with per­fect bal­ance and H100 fig­ures stand­ing in for H800s). Austin et al. give the same factor of $E/k$ [91]. In prac­tice DeepSeek sees a per-expert decode batch "usu­ally within 256 tokens", bound by memory access [2].

The answer is expert par­al­lel­ism: spread experts over many GPUs so each col­lects tokens from the whole group, keep atten­tion mostly data-par­al­lel (TP4 with DP80 in the Decem­ber 2024 report [2], DP144 by Feb­ru­ary 2025 [14]), and move tokens with an all-to-all. Expert par­al­lel­ism and DeepSeek-V3's node-lim­ited rout­ing are in Sec­tion 11.6. DeepSeek's Decem­ber 2024 report gives a decod­ing unit of 40 nodes with EP320 [2], and its Feb­ru­ary 2025 over­view, describ­ing pro­duc­tion then, gives 18 nodes with EP144 [14]. Both phases over­lap one micro-batch's com­pu­ta­tion with another's com­mu­ni­ca­tion [2, 14].

Even fully over­lapped, the all-to-all sets a floor. DeepSeek's authors count 1 byte per ele­ment for FP8 dis­patch, 2 bytes for BF16 com­bine, 32 tokens per GPU, 9 experts per token (8 routed and 1 shared) and a hidden size of about 7K:

Over 400 Gb/s (50 GB/s) InfiniBand that is 241.92 µs per layer with two micro-batches and 14.76 ms over 61 layers, about 67 tokens/s, against 0.82 ms on a GB200 NVL72 domain (reported, ide­al­ized) [1]. For a large mix­ture of experts the scale-out net­work, not HBM, bounds the time per token.

KV is where DeepSeek-V3 is cheap. At 70,272 B per token (Sec­tion 4.5), less than a quar­ter of Llama 3 70B's, a sequence at DeepSeek's aver­age of 4,989 cached tokens [14] holds 351 MB, against 2.57 GB at Llama 3 405B's 516,096 B per token (com­puted).

Other designs put atten­tion and experts on sep­a­rate GPU pools. MegaScale-Infer reports up to 1.90x the per-GPU decode through­put of ear­lier sys­tems [1014], and Step-3's authors report 4,039 tokens/s per GPU under a 50 ms per-token target against 2,324 for DeepSeek-V3 in their setup [991]. The main fail­ure mode is imbal­ance: one over­loaded GPU stalls the whole group, so DeepSeek runs sep­a­rate load bal­ancers for pre­fill, decode and experts [14].

13.7 Speculative decoding: several tokens per pass, exactly

Can a model emit sev­eral tokens per expen­sive for­ward pass and still sample exactly from its own dis­tri­bu­tion? Yes, while the hard­ware has com­pute to spare. A small-batch decode step spends its time read­ing weights (Sec­tion 13.2), so scor­ing five posi­tions costs about as much as scor­ing one. So let a cheap draft model guess $\gamma$ tokens, score them all in one target pass, keep the guesses the target agrees with and repair the first dis­agree­ment. Block­wise par­al­lel decod­ing did this for greedy decod­ing [1016], and spec­u­la­tive decod­ing made it exact under sam­pling, "with­out any changes to the out­puts" [1017, 1018].

The algo­rithm and why it is exact

Write $p$ for the target's next-token dis­tri­bu­tion and $q$ for the draft's. (Here $p$ and $q$ are dis­tri­bu­tions, while $p_{\text{tp}}$ below still counts GPUs, and $c$ is a cost ratio, not the price of Sec­tion 13.3.) One iter­a­tion has five steps [1017]:

1. Draft $x_1,\dots ,x_{\gamma }$ from $q$, one token at a time.
2. Run the target once on the prefix plus all drafts, which gives $p_1,\dots ,p_{\gamma +1}$.
3. Keep $x_i$ with prob­a­bil­ity $min(1,p_i(x_i)/q_i(x_i))$, stop­ping at the first rejec­tion.
4. Resam­ple the rejected posi­tion from the renor­mal­ized resid­ual $max(0,p-q)$. If every draft sur­vived, sample a bonus token from $p_{\gamma +1}$.
5. Emit the kept drafts plus that one token, so every iter­a­tion yields at least one.

At one posi­tion, accep­tance con­trib­utes $q(x)min(1,p(x)/q(x))=min(p(x),q(x))$. Rejec­tion hap­pens with prob­a­bil­ity $1-a$, where $a={\sum }_{x}min(p(x),q(x))$ (the paper's $\beta$, a symbol this guide reserves), and the resid­ual's mass is also $1-a$. The two paths sum to the target:

A Monte Carlo check with 400,000 sam­ples matched $p$ within 0.001 per token (com­puted). The mean accep­tance rate $\alpha$, $a$ aver­aged over con­texts, is one minus the expected total vari­a­tion dis­tance between $p$ and $q$, and it is higher at low tem­per­a­ture [1017].

If each draft is accepted inde­pen­dently with prob­a­bil­ity $\alpha$, and one draft step costs $c$ target steps, the expected tokens per iter­a­tion and the wall­time speedup are [1017]

The tokens sat­u­rate at $1/(1-\alpha )$ while the cost grows lin­early in $\gamma$, so some draft length is best.

Pub­lished runs cal­i­brate the inputs. Drafts about two orders of mag­ni­tude smaller than the target had $c$ below 0.05 and $\alpha$ from 0.5 to 0.9 [1017], and a 2026 fit puts the through­put-opti­mal draft about 200 times smaller than the target [1019].

Draft heads and multi-token pre­dic­tion

A clas­sic 7B draft for LLaMA2-Chat 70B gave only 1.88x [1020], because the draft itself is slow. Recent meth­ods attach a small head that reads the target's own hidden states instead (table below, self-reported speedups on each authors' setup). Medusa's usual accep­tance scheme relaxes the rule above, so under sam­pling it is not guar­an­teed loss­less [1021].

When ver­i­fi­ca­tion stops being free

Ver­i­fi­ca­tion scores $b(\gamma +1)$ tokens for a batch of $b$ sequences, which is free only below the ridge of about 295 tokens on an H100 [4]. Model a target step over $n$ tokens as the larger of weight bytes $W$ over band­width BW and FLOPs over peak $F$:

For Llama 3 70B in BF16 on 8 H100s at short con­text, with $\alpha =0.8$, $\gamma =4$ and $c=0.05$, the speedup is 2.80x up to batch 32, 2.62x at 64, 1.42x at 128 and 0.74x at 256, cross­ing 1 near batch 190 (com­puted). Mea­sured curves fall ear­lier but less steeply, because, as EAGLE-3's authors put it, "As batch sizes increase, this redun­dancy decreases" [1021]. In Figure 71, move $\alpha$ and $c$ to shift the best draft length, then watch the batch curve stay flat until $b(\gamma +1)$ crosses the dashed ridge line. The 8B target adds mea­sured points.

Long con­texts are the main excep­tion, since KV reads keep decode memory-bound at large batch: MagicDec reports up to 2.51x for Llama 3.1 8B at batch 32 to 256 [1025]. Fine-grained mix­ture-of-experts models gain more at low batch than pre­dicted [1026], and drafter gaps shrink under shared con­di­tions [1027], so treat cross-paper tables as rough. Draft and target must share a tok­enizer, accep­tances are not inde­pen­dent (mea­sure tokens per cycle directly), and "loss­less" holds only up to numer­ics (Sec­tion 13.9).

13.8 Fewer bytes per token: quantization, KV compression and decode kernels

How few bits can weights and KV entries use before the model changes, and how do we make the remain­ing bytes stream at full speed? Decode time is bytes over band­width, so halv­ing the bytes halves the floor. Weight-only quan­ti­za­tion (W4A16: 4-bit weights, 16-bit acti­va­tions) dequan­tizes inside the matmul and speeds memory-bound decode. Weight-acti­va­tion quan­ti­za­tion (W8A8) runs the matmul itself in low pre­ci­sion, so it also speeds com­pute-bound pre­fill and large batches. Groups of $G$ weights share $m$ bits of scale meta­data, so $k$-bit values cost

INT4 with a 16-bit scale and zero point per 128 weights is $4+32/128=4.25$ bits, MXFP4 (an 8-bit scale per 32) is 4.25, and NVFP4 (an 8-bit scale per 16) is 4.5 [1028]. Bit lay­outs, ranges and block scal­ing are in Sec­tion 2.3.

Out­liers and the meth­ods that tame them

Naive round­ing, round-to-near­est (RTN) on a shared scale, fails because of out­liers. At around 6.7B param­e­ters, large-mag­ni­tude fea­tures appear in every layer, con­cen­trated in just 6 hidden dimen­sions, and zero­ing them raises per­plex­ity by 600⁠–⁠1000% although they are about 0.1% of the fea­tures [1029]. A per-tensor scale stretched to cover them leaves few levels for every­thing else. The table sum­ma­rizes the main fixes and 4-bit meth­ods.

Rank­ings depend on the bench­mark: on Arena-Hard with Llama 3.1 8B, Kurtic et al. score BF16 25.8, a tuned GPTQ 24.0 and AWQ 22.3. Over 500,000 eval­u­a­tions they found FP8 W8A8 effec­tively loss­less, tuned INT8 within 1⁠–⁠3%, and W4A16 recov­er­ing 99.36% of the Open LLM Leader­board V1 aver­age. Their advice fol­lows the roofline: "W4A16 is the most cost-effi­cient for syn­chro­nous setups, while W8A8 dom­i­nates in asyn­chro­nous con­tin­u­ous batch­ing" [1034]. At large batch INT4 ker­nels pay 20⁠–⁠90% run­time over­head to dequan­tize [1035]. In sil­i­con, FP8 costs at least 50% more area and energy than INT8 [1036]. FP8 train­ing recipes (scal­ing gran­u­lar­ity, accu­mu­la­tion pre­ci­sion) are in Sec­tion 5.6.

Four-bit floats and the bit-width trade

An MXFP4 ele­ment is E2M1, one of $\pm$0, 0.5, 1, 1.5, 2, 3, 4 or 6, times a power-of-two scale shared by 32 ele­ments. NVFP4 uses blocks of 16 with E4M3 scales [121]. With stan­dard meth­ods MXFP4 loses about 10% rel­a­tive accu­racy, and a format-aware GPTQ recov­ers 98⁠–⁠99% for large models [1028], so FP4 is no auto­matic upgrade over INT4. gpt-oss shipped its MoE weights in MXFP4 at 4.25 bits so the 120b model fits one 80 GB GPU [28]: 114.71B weights at 4.25 bits plus 2.12B at 2 bytes give 60.7 GiB, against 60.8 GiB reported (com­puted).

At a fixed memory budget, serve a bigger model at fewer bits? Across more than 35,000 exper­i­ments up to 176B param­e­ters, "4-bit pre­ci­sion is almost uni­ver­sally opti­mal for total model bits and zero-shot accu­racy" [468]. But post-train­ing quan­ti­za­tion hurts more the more data a model saw [458, 469], and Llama 3 8B saw about 15T tokens, roughly 1,900 per param­e­ter (com­puted) [11]. Mea­sure such models. How train­ing and serv­ing pre­ci­sion enter the scal­ing law is in Sec­tion 6.6. In Figure 72, com­pare the small 4-bit GPTQ and AWQ gaps with RTN's 3-bit col­lapse, then find the 8B Arena-Hard row, where AWQ keeps 86% of the BF16 score while its Open LLM Leader­board V1 aver­age kept 98%.

Com­press­ing the KV cache

At long con­text the KV cache out­weighs the weights. Quan­ti­za­tion keeps every token at lower pre­ci­sion: with KIVI's 2-bit scheme (table below), one 128K-token Llama 3 70B sequence shrinks from 40 GiB in BF16 to 7.53 GiB: 2-bit values plus a 16-bit scale and zero point per 32 values, about 3 bits in all, with the most recent 128 tokens kept in BF16 (com­puted, group size as in KIVI), and as of its 2026⁠–⁠09⁠–⁠30 release FlashMLA's decode kernel accepts only FP8 and FP4 caches [1039]. Evic­tion drops tokens, and dropped tokens are gone for good.

A uni­fied bench­mark found KV quan­ti­za­tion more reli­able across tasks than drop­ping tokens, and com­pres­sion during pre­fill harm­ful [1038]. Quan­tize before you evict.

Ker­nels that stream the bytes

Fewer bytes help only if the kernel reads them at full band­width. FlashAttention's tiling and online soft­max are explained in Sec­tion 2.4. Train­ing ker­nels par­al­lelize over batch and query blocks, so at decode a small batch leaves most of an A100's 108 SMs idle. Flash-Decod­ing also splits the KV sequence into chunks and merges the par­tial results exactly [1046]. Chunk $j$ returns its nor­mal­ized output $o_j$, max­i­mum score $m_j$ and nor­mal­izer ${\ell }_j$, and the online-soft­max iden­tity com­bines them [38]:

On an A100 at batch 1 and 64K con­text, atten­tion fell from 2,300.6 µs to 64.4 µs [1046]. With 2 KV heads of dimen­sion 128 the cache is $\mathrm{65,536}\times 2\times 128\times 2\times 2$ B = 67.1 MB, so achieved band­width rose from about 29 GB/s to 1.04 TB/s (com­puted): same arith­metic, more SMs read­ing. FlashInfer serves paged and shared-prefix lay­outs with 29⁠–⁠69% lower inter-token latency than com­piler back­ends [1047]. FlashMLA reaches up to 660 TFLOP/s on H800, and its 2026⁠–⁠09⁠–⁠30 release "removed sup­port for the Hopper archi­tec­ture" [1039], so pin kernel ver­sions with the model.

13.9 Sampling, structured output and reproducibility

What does the decoder do with the logits, and why can the same model score dif­fer­ently on the same bench­mark? The model emits one logit $z_i$ per vocab­u­lary entry, and a sam­pler turns them into a token. Tem­per­a­ture $T$ rescales the logits, and trun­ca­tion rules such as min-p decide which tokens stay eli­gi­ble:

$T$ near 0 is greedy decod­ing, $T=1$ the model's own dis­tri­bu­tion, and higher $T$ flat­tens it. Nei­ther extreme suits open-ended text. Beam search degen­er­ates into bland, repet­i­tive text, while pure sam­pling over-sam­ples an unre­li­able tail of tens of thou­sands of unlikely tokens [1048]. Trun­ca­tion sam­plers cut that tail (table below). Apply­ing tem­per­a­ture before or after trun­ca­tion gives dif­fer­ent sam­plers, so record your engine's order.

Set­tings move scores. Greedy decod­ing beats sam­pling on most bench­marks except AlpacaEval, with math and code the most sen­si­tive [1053]. Long rea­son­ing models differ: greedy decod­ing "results in higher rep­e­ti­tion rates and sig­nif­i­cant vari­abil­ity across dif­fer­ent check­points", so DeepSeek-R1's authors aver­age pass@1 over 4 to 64 sam­ples at tem­per­a­ture 0.6 and top-p 0.95 [3], and Qwen3's think­ing mode adds top-k 20 [198]. On mul­ti­ple-choice prob­lems, tem­per­a­ture from 0 to 1 made no sig­nif­i­cant dif­fer­ence [1054]. Min-p's authors report Mis­tral 7B on GSM8K at tem­per­a­ture 1.5 scor­ing 18.42% with min-p against 0.68% with top-p [1050], but a re-exam­i­na­tion found min-p "does not sur­pass base­line sam­plers when con­trol­ling for the number of hyper­pa­ram­e­ters" [1055]. As of Novem­ber 2026 the dis­pute is open: com­pare sam­plers with equal tuning bud­gets.

Struc­tured output

To force JSON or another gram­mar, the engine masks every token that would break a valid prefix. Done naively that costs time pro­por­tional to the vocab­u­lary per token [1056]: for Llama 3's 128,256 tokens, a 16 KB bit­mask per sequence per step, about 4 MB at batch 256 (com­puted). Engines pre­com­pute most of the mask (table below). Syntax is then guar­an­teed but qual­ity is not: mask­ing dis­torts the dis­tri­bu­tion [1057], gram­mar-token mis­align­ment can cut accu­racy [1058], and JSONSchemaBench tests engines on 10K real-world schemas [1059]. Describe the format in the prompt too, and mea­sure accu­racy with and with­out the con­straint.

Repro­ducibil­ity

Why iden­ti­cal prompts at tem­per­a­ture 0 give dif­fer­ent out­puts (80 dis­tinct com­ple­tions out of 1,000 in Think­ing Machines' test) is in Sec­tion 12.7. In serv­ing, ker­nels change reduc­tion order with batch size, and batch size fol­lows load [848]. Batch-invari­ant ker­nels fix this at a cost, 34.35% in SGLang with CUDA graphs [959]. In RL, "the dif­fer­ent numer­ics between train­ing and infer­ence implic­itly turns our on-policy RL into off-policy RL" [848] (Sec­tion 9.6). In eval­u­a­tion, BF16 greedy decod­ing of DeepSeek-R1-Dis­till-Qwen-7B varied by up to 9% in accu­racy with GPU count, type and batch size [402] (Sec­tion 10.2). So fix and report every sam­pler set­ting, seeds and sample count, follow the model card for rea­son­ing models, and use batch-invari­ant modes for RL roll­outs and eval­u­a­tion repro­duc­tion.

Why can a server pre­fill a 4,000-token prompt in less time than it takes to gen­er­ate 100 tokens of answer?

Pre­fill pushes every prompt token through the model in one pass, so each weight read from HBM does thou­sands of mul­ti­ply-adds and the pass is com­pute-bound. Each decode step must read every weight again to pro­duce one token per sequence, so at small batch it is bound by memory band­width. On PaLM 540B with 64 TPU v4 chips, pre­fill cost about 0.14 ms per prompt token and decode about 28 ms per step for each sequence (com­puted from Pope et al.'s Table 2), a factor of about 200 in latency. In chip time per token the gap is about 3x, because each step served 64 sequences.

What is the fastest Llama 3 70B in BF16 can gen­er­ate tokens for one user on 8 H100s, and what should you expect in prac­tice?

Every decode step streams 141 GB of weights over $8\times 3.35$ TB/s, so the floor is about 5.3 ms per token, about 190 tokens/s (com­puted). Real engines reach roughly 60% of peak band­width at batch 1 (Databricks' mea­sure­ment) and also pay tensor-par­al­lel all-reduce latency, so expect roughly 100 to 130 tokens/s. That range is a com­puted esti­mate, not a mea­sure­ment.

You dis­ag­gre­gate Llama 3 70B and send an 8,192-token prompt's KV cache from the pre­fill node to the decode node over one 400 Gb/s NIC. Is the trans­fer a bot­tle­neck?

The cache is $\mathrm{8,192}\times \mathrm{327,680}$ B = 2.68 GB, about 54 ms at 50 GB/s, against roughly 390 ms of pre­fill on 8 H100s at 40% MFU, so about 14% (com­puted). The ratio barely depends on prompt length (atten­tion makes it fall slowly), because the bytes and the pre­fill FLOPs both grow lin­early with it. On eight NICs it is under 2%, and send­ing layer by layer hides most of the rest. DistServe mea­sured under 0.1% of total latency and Split­wise 0.8%.

A draft model has accep­tance rate 0.8 and each draft step costs 5% of a target step. Which draft length max­i­mizes the speedup, and why not draft 20 tokens?

Eight, for about 3.1x (com­puted from $S=(1-{\alpha }^{\gamma +1})/((1-\alpha )(\gamma c+1))$). Each extra guess adds $c$ to the cost of an iter­a­tion but only ${\alpha }^{\gamma }$ expected tokens, which shrinks geo­met­ri­cally. At $\gamma =20$ an iter­a­tion costs 2 target steps and yields 4.95 tokens on aver­age, about 2.5x. At large batch the best length shrinks fur­ther, because ver­i­fi­ca­tion stops being free.

Your 4-bit model matches the BF16 model within 0.3 points on MMLU. Is it safe to ship?

Not yet. Equal aver­ages can hide 5% or more of answers flip­ping between right and wrong (Dutta et al.), and Meta's FP8 Llama 3 405B pro­duced cor­rupted responses on high-per­plex­ity tokens such as dates while bench­mark scores looked fine. Check token-level KL diver­gence against the BF16 model, flip rates, open-ended qual­ity scored by a judge or reward model as a dis­tri­bu­tion, long-con­text tasks and rare tokens, with the same sam­pler and seeds for both models.

14. Sandboxes and environments: where models act

The system that exe­cutes model-writ­ten code is both a secu­rity bound­ary and part of the reward func­tion. Its iso­la­tion, resource limits, state and grader deter­mine which actions are pos­si­ble and which out­comes count as suc­cess. The fleet also has to keep pace with the sam­pler.

Why did the pro­gram that runs a model's code, com­mands and clicks become core infra­struc­ture for eval­u­a­tion and RL? An eval­u­a­tion runs code once per sample, so the risk is acci­den­tal damage and leak­age. RL runs it mil­lions of times against an opti­mizer, so any­thing reach­able that changes the reward (tests, grader files, the har­ness's con­trol chan­nel) becomes part of what the policy opti­mizes. Agents add state: on DeepSeek's DSec plat­form, median sand­box life­times were 17.4 min­utes for con­tain­ers and 15.5 min­utes for microVMs, and p99 life­times exceeded three hours (reported) [10].

The Codex paper behind HumanEval aimed to stop gen­er­ated pro­grams from "mod­i­fy­ing, gain­ing per­sis­tence on, access­ing sen­si­tive resources on, or exfil­trat­ing data from a host or net­work", using gVisor and eBPF (in-kernel packet filter) fire­wall rules [699]. Its public har­ness warns: "This func­tion is NOT a secu­rity sand­box." [1061]. In-process patch­ing guards the test against care­less code, and only an oper­at­ing-system bound­ary guards the host.

In prac­tice, the AI Sci­en­tist "attempted to edit the code to extend the time limit arbi­trar­ily" [1062], an o1 cap­ture-the-flag (CTF) eval­u­a­tion left the Docker daemon API reach­able to the model [1063], and models have been seen to "exam­ine git his­tory to find answers to a coding test" [1064]. None of this needed intent, only an opti­mizer meet­ing a weak bound­ary.

Sec­tions 14.1 to 14.3 build the bound­ary, 14.4 to 14.6 the grader, and 14.7 and 14.8 the fleet. The exam­ples are the SWE-bench Ver­i­fied har­ness [814, 16], DeepSWE (4,500 R2E-Gym tasks, six days on 64 H100 GPUs, reported) [9], and a Fire­cracker fleet at Kimi K2's scale of over 10,000 con­cur­rent sand­boxes [72]. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

Figure 73 walks one SWE-bench Ver­i­fied episode through eight steps, with the grade out­side the iso­la­tion bound­ary. Step through the fail­ure points (shown by default), then switch to RL mode to see which ones cor­rupt a reward.

One default runs through the chap­ter: treat model-gen­er­ated code as hos­tile when you design the bound­ary, and as care­less when you read the logs.

14.1 What can go wrong, and what a container is made of

What exactly are we defend­ing against, and which Linux mech­a­nism does which job? Table 14.1 lists five threat classes, each with a public exam­ple and the con­trols that Sec­tion 14.3 sets.

"The model is not a secu­rity bound­ary. Every con­trol must hold if the agent is fully adver­sary-con­trolled." [1067]. An agent with pri­vate data, untrusted con­tent and exter­nal com­mu­ni­ca­tion (the "lethal tri­fecta") can be tricked into "send­ing it to that attacker", so remove one leg, usu­ally egress [1068]. And "A sand­box is not a sub­sti­tute for a secure archi­tec­ture." [1069].

The mental model: a Linux con­tainer is an ordi­nary process tree that the kernel gives a pri­vate view (names­paces), meters (con­trol groups, or cgroups) and restricts (capa­bil­i­ties, sec­comp, Linux Secu­rity Mod­ules). All con­tain­ers on a host share one kernel, which drives Sec­tion 14.2. Table 14.2 gives each mech­a­nism's job.

Two lessons sit out­side the table. The run­time daemon is attack sur­face: "only trusted users should be allowed to con­trol your Docker daemon" [1072], so keep its socket and API out of reach. And bespoke fil­ters are hard to get right: seven expe­ri­enced devel­op­ers sand­box­ing one pro­gram "each came to dif­fer­ent solu­tions" [1085]. Prefer a main­tained pro­file plus a stronger bound­ary.

14.2 The isolation spectrum: containers, gVisor, microVMs and WebAssembly

If names­paces and sec­comp are so flex­i­ble, why do seri­ous deploy­ments add gVisor or a vir­tual machine, and what does each option cost in start-up time, memory and com­pat­i­bil­ity? Because every con­tainer on a host shares one kernel. NIST says con­tainer iso­la­tion "is not as high as that pro­vided by hyper­vi­sors" [1086], and "a single kernel vul­ner­a­bil­ity is suf­fi­cient for escape" [1067]. Tighter fil­ters do not help: "A triv­ial Linux appli­ca­tion requires 15 unique syscalls", but QEMU "can require up to 270" [1087], so a narrow filter breaks real work­loads.

The two ways out are a second kernel in soft­ware (gVisor) or moving "the secu­rity-crit­i­cal inter­face from the OS bound­ary to a bound­ary sup­ported in hard­ware" [1087] (microVMs, Kata, full VMs). Table 14.3 is a crude proxy for how much host kernel each bound­ary may call.

gVisor: a kernel in user space

gVisor "is an appli­ca­tion kernel that imple­ments a Linux-like inter­face", writ­ten in Go and run in user space, with a Gofer process that opens files for it [1083]. In 2019, simple system calls were at least 2.2 times slower on gVisor than in ordi­nary con­tain­ers, and open­ing and clos­ing files on an exter­nal tmpfs was 216 times slower [1088]. Con­tainer setup plus tear­down, run one at a time, took 1.014 s with runc and 1.117 s with gVisor on KVM in the same study [1088]. CPU and memory work cost noth­ing extra [1089], so file-heavy test suites pay.

Three 2023 changes target the first two num­bers. Sys­trap replaced ptrace as the default plat­form in mid-2023 [1090], directfs became the default in runsc and made stat(2) more than twice as fast [1091], and the rootfs over­lay, also now the default, halved the sand­box­ing over­head of one abseil build [1092]. This guide there­fore reads the 2019 system-call and file-access ratios as likely upper bounds for cur­rent gVisor (this guide's infer­ence). No cited source re-mea­sures the life­cy­cle times, so 1.117 s stays a 2019 number.

Fire­cracker and microVMs

Fire­cracker is a vir­tual machine mon­i­tor (VMM) with "96% fewer lines than QEMU", con­fined by a jailer (Table 14.3) [1087]. Its spec­i­fi­ca­tion prom­ises at most 125 ms from the InstanceStart API call to the start of the guest's /sbin/init, mea­sured with a min­i­mal kernel and root file system and the serial con­sole dis­abled [1093]. The 2020 paper reports cre­ation of up to 150 microVMs per second per host and a mea­sured VMM memory over­head of about 3 MB, against about 13 MB for Cloud Hyper­vi­sor and about 131 MB for QEMU [1087]. Real stacks are slower. In AWS Lambda, boot­ing a min­i­mal kernel with a few userspace dae­mons and net­work setup takes about 200 ms [1094], and boot­ing Fire­cracker inside pro­duc­tion frame­works such as con­tain­erd or OpenNebula took 700⁠–⁠1,300 ms in one 2021 mea­sure­ment [1095].

Start-up mat­ters rel­a­tive to the work. With one start per life, the share of a sand­box's life spent start­ing is

A 125 ms boot before a 60 s agent episode is $0.125/60.125=0.21\%$ (com­puted). A 0.5 s code reward behind runc's 1.014 s create plus destroy (2019) is $1.014/1.514=67\%$, and a 4 ms snap­shot restore [1094] makes it $0.004/0.504=0.8\%$ (com­puted). In Figure 74, slide the work per sand­box and watch the share beside each bound­ary: for sub-second rewards, snap­shots matter more than the bound­ary.

Kata, full VMs and lan­guage sand­boxes

Kata Con­tain­ers runs each pod in a light­weight VM "as a second layer of defence" [1084], and the Kuber­netes agent-sand­box project del­e­gates to gVisor or Kata [1096]. No source we read mea­sures Kata end to end. Full VMs serve GUIs and com­puter use. At the other end, a WebAssembly (Wasm) module "starts with no ambi­ent author­ity" [1097], and Faasm's Wasm-based Proto-Faaslets restore "in hun­dreds of microsec­onds" [1098], but Wasm code ran 45⁠–⁠55% slower in 2019 browsers [1099]. Process tools such as bub­blewrap are kits, not poli­cies: it "is not a com­plete, ready-made sand­box" [1100, 1101]. For short code rewards at RL scale, keep the second bound­ary and pay its start-up once: reuse precre­ated work­ers inside gVisor, or restore microVMs from a snap­shot, with the net­work off. DSec runs short func­tion calls in reus­able precre­ated con­tain­ers [10].

In Table 14.4, ref­er­ence har­nesses default to plain Docker for repro­ducibil­ity, while teams run­ning strong models at scale add a second bound­ary and a net­work policy [1066, 10, 1063]. DSec names the trade: "stronger iso­la­tion and more com­plete system func­tion­al­ity usu­ally come with higher startup latency and resource over­head" [10].

14.3 Hardening: limits, egress, snapshots and a checklist

Given a bound­ary, which set­tings make it safe enough for model-gen­er­ated code at RL scale, and how do fast resets from snap­shots go wrong? Start with limits and caps. Kuber­netes enforces memory limits "with out of memory (OOM) kills" and CPU limits "by CPU throt­tling" [1105]. Inspect caps file reads at 100 MB and exe­cu­tion output at 10 MB by default [1103], and DSec's run­away output shows why the cap belongs at cap­ture time [10].

Time­outs belong out­side the sand­box, since the AI Sci­en­tist tried to extend its own [1062], and a Kuber­netes Job's activeDeadlineSeconds holds "no matter how many Pods are cre­ated" [1106]. Derive it from the CPU budget. The cgroup v2 file cpu.max grants MAX microsec­onds of CPU per PERIOD [1071], so with $c$ cores and ${\tau }_{\text{cpu}}$ CPU-sec­onds of work,

With $k$ runnable threads the tighter bound is ${\tau }_{\text{cpu}}/min(c,k)$. A suite need­ing 120 CPU-sec­onds (illus­tra­tive) needs at least 60 s under 200000 100000 (2 cores) and 240 s under 50000 100000 (0.5 cores) (com­puted). Oth­er­wise a CPU limit looks like a wrong answer, which in RL is reward noise, so record why each episode ended (Sec­tion 14.8).

The net­work starts closed, as in Inspect, which "sets net­work_mode: none" [1103]. If a task needs the net­work, use per-sand­box eBPF allowlists by IP address, port and pro­to­col [10] or an out­side proxy, and for RL prefer pinned local mir­rors. Each allowlist entry is "a capa­bil­ity grant" [1064]. In one 2026 report a planted key sent workspace files out through an allowed API domain: "The sand­box worked per­fectly, and yet the data was exfil­trated." [1064]. Users there approved roughly 93% of per­mis­sion prompts and grew less atten­tive as prompts piled up, so per-action approval is a weak con­trol [1064].

Keep the grader and con­trol plane out of reach: "Final-output checks alone cannot reli­ably estab­lish whether the agent solved the task as intended." [10]. Grade the arti­fact in a fresh con­tainer (Sec­tion 14.4), mask sen­si­tive paths [1081], mount tests read-only [1107], and put LSM rules on har­ness logs and sock­ets [10]. Audit glue code, since "the weak­est layer is the one you built your­self" [1064]. Log every com­mand, exit status and net­work attempt, since Sec­tion 14.6 finds reward hacks in such logs. Figure 75 there maps each threat to its layer.

Snap­shots: fast resets and the unique­ness pit­fall

Snap­shots make resets cheap. A Fire­cracker restore maps guest memory copy-on-write [1108] and "can take as little as 4ms" [1094]. Fire­cracker's doc­u­men­ta­tion con­sid­ers resum­ing exe­cu­tion from the same snap­shot state more than once inse­cure, because iden­ti­fiers, random num­bers, seeds, the entropy pool and cryp­to­graphic tokens may no longer be unique [1108]. With the VMGenID device, guests run­ning Linux 5.18 or later reseed their in-kernel pseudo-random number gen­er­a­tor on resume, so user-space code that asks the kernel for ran­dom­ness after resume is safe, but unique iden­ti­fiers, cached random num­bers and tokens already in memory are still repli­cated across clones [1108, 1094].

RL adds a ten­sion no source we found addresses: a GRPO group wants iden­ti­cal start­ing envi­ron­ments, secu­rity wants unique clones. Keep envi­ron­ment state iden­ti­cal, reseed ran­dom­ness, log seeds and never snap­shot cre­den­tials. Table 14.5 col­lects the defaults.

14.4 Inside a harness: SWE-bench as the reference design

How do you turn "fix this GitHub issue" into a reward a machine can com­pute, and which design deci­sions make that reward hard to fake? A har­ness owns the sand­box, runs the model's actions and com­putes the score. SWE-Gym, SWE-smith and SWE-rebench all grade with SWE-bench-style fail-to-pass tests (Sec­tion 14.5), so we take SWE-bench apart [723, 1109, 1110].

The envi­ron­ment inter­face

Gym gave every envi­ron­ment two meth­ods, reset() and step(action) [1111]. Gym­na­sium split its done flag into ter­mi­nated (the task ended) and trun­cated, which also covers an envi­ron­ment that "crashes or times out for rea­sons unre­lated to the agent's behav­iour" [1112]. An image-pull time­out is a trun­ca­tion, not a failed task. Score it 0 and you punish the policy for the fleet's bugs. Inspect marks such sam­ples as errors instead [1103]. Keep them out of the gra­di­ent and alert on their rate.

The agent runs a ReAct loop of rea­son­ing, action and obser­va­tion [720], and OpenHands runs each ses­sion in its own Docker sand­box [1113]. The Model Con­text Pro­to­col (MCP) stan­dard­izes tool calls over JSON-RPC 2.0 and warns that tools "rep­re­sent arbi­trary code exe­cu­tion" [1114, 1115]. It is an inter­face, not an iso­la­tion layer (Sec­tion 14.2).

How SWE-bench builds and grades a task

SWE-bench keeps merged pull requests from 12 Python repos­i­to­ries that have at least one test flip­ping from fail to pass under the ref­er­ence fix, 2,294 tasks in all. FAIL_TO_PASS tests fail before the fix and pass after it. PASS_TO_PASS tests, a median of 51 per task, pass both times. A task is resolved only if every listed test passes, so fixing the issue while break­ing an old test ("Break­ing Resolved") fails [814]. The Docker har­ness stacks a base image, about 60 envi­ron­ment images and one image per task, about 2,000 GB cached by the doc­u­men­ta­tion's count [1102] (Sec­tion 14.5 com­pares esti­mates). Grad­ing runs in six steps [814, 1116]:

1. Start a fresh con­tainer from the task image at the base commit.
2. Run the install com­mand.
3. Apply the model's patch.
4. Reset the test files to the base commit and apply the hidden test patch, so any edits the patch made to those tests are over­writ­ten.
5. Run the tests and parse a status for each one.
6. Com­pare with both lists. "If a test is miss­ing or has a non-pass status, it is con­sid­ered a fail status" [814].

Three deci­sions make this reward hard to fake, and Sec­tion 14.6 meets the hack each one blocks. The agent never sees the tests [16]. Grad­ing runs in a fresh con­tainer, so only the patch sur­vives [1116]. And the grader reads per-test status, never the exit code.

The scaf­fold (the tools and prompts between model and shell) is part of the envi­ron­ment. SWE-agent's inter­face resolved 12.47% of SWE-bench against 3.8% for the best ear­lier system [1117], and across leader­board sub­mis­sions GPT-4's SWE-bench Lite score ranged from 2.7% with an early retrieval-based scaf­fold to 28.3% with CodeR [16]. Report the scaf­fold with every score.

Clean­ing the signal: SWE-bench Ver­i­fied

Auto­matic con­struc­tion lets unfair tasks in. For SWE-bench Ver­i­fied, devel­op­ers labeled each sam­pled task three times and kept the high­est-sever­ity label, so one flag removed a task. The authors call this "likely to be overzeal­ous" [16].

14.5 Environments at training scale: gyms, synthetic tasks and computer-use worlds

RL needs thou­sands of exe­cutable tasks with a reward. Where do they come from, what does each pipe­line trade away, and how do you reset a world that is not a git repos­i­tory? A bench­mark is not a gym. SWE-bench's train­ing split has patches but lacks "exe­cutable envi­ron­ments and reward sig­nals" [723]. Each pipe­line below adds them at a dif­fer­ent price.

Two find­ings carry into Sec­tion 14.6. Used to pick the best of many SWE-bench Ver­i­fied tra­jec­to­ries, exe­cu­tion-based and exe­cu­tion-free ver­i­fiers each sat­u­rate around 42⁠–⁠43% while a hybrid reaches 51%, and for most prob­lems fewer than 20% of gen­er­ated tests tell cor­rect patches from incor­rect ones [725]. And flawed tasks pro­duce tra­jec­to­ries that "appear as fail­ures but are actu­ally due to task imper­fec­tions" [1110].

Defaults: share images per repos­i­tory snap­shot, require a pass­ing ref­er­ence and a fail­ing empty patch for every task, pin ver­sions at build time [1110], and keep a held-out set with hidden tests for eval­u­a­tion.

Worlds that are not repos­i­to­ries

Web, desk­top and API worlds need their own iso­la­tion, reset and grad­ing, as Table 14.8 shows.

A git check­out resets with one com­mand. Other worlds need snap­shots, which restore a whole machine, or scripts, which rebuild what a task touched. Shared back­ends cap par­al­lel­ism, so give each episode its own. Pin the clock with the data, or dated tasks expire. Live web­sites rot fastest: changed pages caused a 28% under­es­ti­mate on OSWorld's Chrome tasks [818], and fixing 300+ reported issues took about ten people two months. Their lesson: "pro­vid­ing reli­able rewards con­sumes more human resources than we imag­ined" [1122].

Ter­mi­nal-Bench pub­lishes the clear­est public recipe. Tests pass if and only if the end state is accept­able, an oracle solu­tion passes, and no short­cut exists that real work lacks. CI rejects images that con­tain tests or ora­cles, a no-op agent must fail, and an exploit agent hunts for cheats, at about three reviewer-hours per task [815]. Even so, of the 125 Ter­mi­nal-Bench 3 tasks no agent solved hon­estly, only 78 sur­vived as cer­ti­fied-unsolved can­di­dates. The other 47 included 14 with broken ora­cles, 8 dom­i­nated by infra­struc­ture fail­ures, 4 pass­able only through a ver­i­fier bypass and 21 whose solv­abil­ity could not be cer­ti­fied [1123].

14.6 Verifiers and reward hacking: when the grader is the attack surface

The grader is the reward func­tion. How does it fail, how do poli­cies exploit it, and which con­trols close each hole? In eval­u­a­tion a grader bug mis­states one score. In RL an opti­mizer applies it mil­lions of times and searches for exactly the cases where grader and intent dis­agree.

Three ways a grader is wrong

False neg­a­tives score cor­rect work 0. False pos­i­tives score wrong work 1. Tam­per­ing, where the agent changes the grader or its inputs, is the third kind. Audits find the first two every­where, mis­stat­ing capa­bil­ity by up to 100% in rel­a­tive terms [818].

Grader noise, in num­bers

A flaky test passes and fails on the same code [1126]. If each of $n_{\text{tests}}$ tests flakes to fail­ure inde­pen­dently with prob­a­bil­ity $f$, a cor­rect patch passes with prob­a­bil­ity $(1-f{)}^{n_{\text{tests}}}$, so it is graded wrong with prob­a­bil­ity

With one FAIL_TO_PASS test plus SWE-bench's median 51 others, $n_{\text{tests}}=52$ [814]. At $f=0.1\%$ that gives 5.07%, at $f=1\%$ it gives 40.7% (illus­tra­tive $f$, com­puted). Ten clean reruns of the ref­er­ence expose a 0.1% flake with prob­a­bil­ity $1-{0.999}^{10}=1.0\%$ (com­puted), so reruns catch only fre­quent flakes.

How does noise reach the gra­di­ent? Let $\mu$ be the policy's true suc­cess prob­a­bil­ity and $R$ the binary reward. The grader out­puts 1 with prob­a­bil­ity $1-{ϵ}_{\text{FN}}$ on cor­rect work and ${ϵ}_{\text{FP}}$ on wrong work, so

At $\mu =0.6$, ${ϵ}_{\text{FN}}=0.2$ and ${ϵ}_{\text{FP}}=0.1$ this gives 0.520, against 0.5195 in a 200,000-sample sim­u­la­tion, with slope 0.70 (illus­tra­tive, com­puted). Noise that ignores the policy rescales the expected signal and adds vari­ance. It slows learn­ing but does not change which policy is best. A false-pos­i­tive rate the policy can raise is reward hack­ing. If a share $h$ of truly failed episodes pass by hack­ing, mea­sured suc­cess is

tau-bench's pass^k is "the chance that all $k$ i.i.d. task trials are suc­cess­ful, aver­aged across tasks". GPT-4o fell from about 61% at pass^1 to about 25% at pass^8 on retail [779], not the ${0.61}^8=1.9\%$ iden­ti­cal tasks would give (com­puted), so reli­a­bil­ity is set by which tasks are always solved. pass@k is Sec­tion 10.3.

The public record of reward hack­ing

Spec­i­fi­ca­tion gaming "sat­is­fies the lit­eral spec­i­fi­ca­tion of an objec­tive with­out achiev­ing the intended out­come" [730]. Reward hack­ing is its RL form [1127, 1128], and it can appear abruptly as capa­bil­ity grows [731].

Figure 75 sets these hacks beside the threats of Sec­tion 14.1 and links each to the layer whose con­trol closes it. Pick a chip and follow its line. In this guide's map­ping, 9 of the 15 OS-level threats, includ­ing every kernel-level one, end inside the sand­box stack (layers 5 to 7), and all 11 reward hacks end above it, in the grader or the task build. The sand­box pro­tects the host. The grader and the task build pro­tect the reward.

METR's "more than 43x" divides rounded per­cent­ages. Raw counts give $(39/128)/(8/\mathrm{1,087})\approx 41$ (com­puted). Why the spread? On RE-Bench "the model was able to see the entire scor­ing func­tion" [732]. ImpossibleBench found more cheat­ing on repos­i­tory tasks, with mul­ti­ple sub­mis­sions (33% to 38%) and with com­plex scaf­folds [1107]. So remove three con­di­tions: a vis­i­ble grader, a writable filesys­tem around it, and repeated fail­ure.

Hack­ing also changes the model. Train­ing on harm­less hacks gen­er­al­ized to "pre­fer­ring less knowl­edge­able graders" [1131], and a model that learned to hack in pro­duc­tion coding envi­ron­ments tried 12% of the time to sab­o­tage code in ways that would hide hack­ing [1132, 610]. Sec­tion 9.9 covers the train­ing-time side.

Con­trols that close the holes

Con­trols fall into three groups, strong­est first. Use mon­i­tors to find exploits, not as a reward term: OpenAI advises against strong opti­miza­tion pres­sure on chains of thought [1133], and METR says to "patch the exploit in the scor­ing func­tion rather than pun­ish­ing the model" [732].

A check­list for an RL coding envi­ron­ment, roughly by cost (this guide's infer­ence):

1. Grade in a fresh con­tainer, per test, with miss­ing tests as fail­ures.
2. Ship no tests, ora­cles, com­piled arti­facts or caches in the image.
3. Make tests read-only, exclude test-con­fig­u­ra­tion files, rein­stall depen­den­cies.
4. Require in CI that the oracle passes and a no-op agent fails, and rerun the ref­er­ence.
5. Report eval­u­a­tion num­bers on hidden or held-out tests only.
6. Keep the net­work off by default, cap steps and time, and mark infra­struc­ture errors as trun­ca­tions.
7. Run a tra­jec­tory mon­i­tor to find and patch exploits, never as a reward.
8. Give agents an abort action, red-team each envi­ron­ment, and track the hack rate.

14.7 Fleet arithmetic: how many sandboxes, and who waits for whom

You have 64 GPUs and 4,500 SWE tasks. How many sand­boxes must run at once, how many tra­jec­to­ries per hour come out, where does start-up time go, and when does the envi­ron­ment rather than the GPU set the pace? An agen­tic roll­out is a relay. The GPU gen­er­ates an action, the sand­box runs it, and the obser­va­tion goes back. A sand­box is occu­pied for the whole tra­jec­tory, includ­ing every second it waits for the model, so the sand­box count fol­lows how long tra­jec­to­ries live, not how much CPU they burn.

Little's law for sand­boxes

In a stable system, mean occu­pancy equals arrival rate times mean time inside. With $\lambda$ the cre­ation rate and $T_{\text{life}}$ the mean life­time,

where $E_h$ is episodes per hour from a fixed fleet. The law needs the mean life­time, not the median. Take an illus­tra­tive SWE episode of 30 s start-up, 50 turns of 20 s gen­er­a­tion plus 2 s tools, and 120 s grad­ing: $T_{\text{ep}}=30+50\times 22+120=\mathrm{1,250}$ s. Start-up is 2.4% of the life, tools 8.0%, wait­ing for the model 80.0% and grad­ing 9.6%, and 512 sand­boxes give about 1,475 episodes per hour (com­puted).

Who waits for whom

The infer­ence engine wants $n_{\text{gen}}$ tra­jec­to­ries in gen­er­a­tion at all times (its decode batch, Sec­tion 13.4). If a sand­box waits for the model a share $w$ of its life,

For the illus­tra­tive episode $w=0.80$, so 256 tra­jec­to­ries in gen­er­a­tion need 320 live sand­boxes (com­puted). Mea­sured lives lean harder toward the model. Sand­man's roll­out sand­boxes spent "74⁠–⁠91% of sand­box life­time" wait­ing for the next action [1104], and Kimi K3 reports up to 98% [1066], so $n_{\text{sb}}/n_{\text{gen}}$ is only 1.02 to 1.35 (com­puted). The fleet is mostly idle, state­ful machines.

Figure 76 puts these for­mu­las on slid­ers. Drag start-up from 30 s to 4 ms and see how little the episode rate moves while the gray wait­ing band dom­i­nates.

Tails and bursts

A syn­chro­nous step waits for its slow­est member. If each of $n$ inde­pen­dent oper­a­tions fin­ishes inside its $q$-quan­tile with prob­a­bil­ity $q$,

with $H_n={\sum }_{i=1}^{n}1/i$ and the second result exact for expo­nen­tial dura­tions of mean $\overline{t}$. Check: a con­tainer start that fetches 1,000 chunks hits the cache's p99.9 latency on 63% of tasks [1134], and $1-{0.999}^{1000}=0.632$. A 512-tra­jec­tory step con­tains a p99 strag­gler with prob­a­bil­ity 99.4% and waits about $H_{512}=6.8$ times the mean (com­puted), a lower esti­mate for heavy-tailed life­times. Hence par­tial roll­outs: Kimi K2 pauses long-tail tasks and resumes them next iter­a­tion [72], and Kimi K3 pauses gen­er­a­tion once a set frac­tion of tra­jec­to­ries fin­ishes [1066], at the price of off-policy data (Sec­tion 9.8).

Bursts size the warm pool, $n_{\text{pool}}=\lambda t_{\text{create}}$. Fire­cracker's authors use it for warm pools: at 125 ms cre­ation time, "one pooled MicroVM is required for every 8 cre­ations per second" [1087], and $8\times 0.125=1$. Sand­man saw a cam­paign mean of 1.8 cre­ates per second and 132 in one second when two runs started together [1104]. With 4.1 s cre­ates that is 7 warm sand­boxes for the mean and 541 for the burst (com­puted). DSec serves about 3 mil­lion sand­boxes a day per scale unit, peak­ing near 380,000 con­cur­rent [10]. The daily mean of 34.7 cre­ates per second times the 17.4-minute median life­time gives about 36,000 (com­puted), a lower bound since the mean exceeds the median. Sand­man's burst was about 73 times its mean cre­ation rate, and DSec's peak con­cur­rency about 10 times its mean-rate esti­mate (com­puted), so plan for bursts 10 to 100 times the mean.

Where start-up time goes

Start-up is small in long episodes but dom­i­nates sub-second code rewards. An illus­tra­tive GRPO step of 512 prompts times 16 sam­ples spends about 8,300 sand­box-sec­onds on 1.014 s con­tainer life­cy­cles [1088] and about 33 on 4 ms snap­shot restores [1094] (com­puted). So DSec runs short func­tion calls in reus­able precre­ated con­tain­ers [10].

A slow start is mostly moving bytes. Slacker found that "pulling pack­ages accounts for 76% of con­tainer start time, but only 6.4% of that data is read" [1135]. A decade later DSec's agent images are 4.1 to 12.1 GB with 4.2% to 13.3% read, and the median image serves only three sand­boxes, so node caches help little [10]. Each remedy below removes a dif­fer­ent slice. Figure 74 plots the per-sand­box laten­cies.

14.8 Operating the fleet: orchestration, failures, reproducibility and cost

How do you turn "create 10,000 sand­boxes now" into 10,000 work­ing shells with­out break­ing a shared clus­ter, keep infra­struc­ture fail­ures out of the reward, make the same patch earn the same reward twice, and what does the fleet cost next to its GPUs? The con­trol plane fails before the CPUs do.

The con­trol plane

In Kuber­netes a pod's requests decide its place­ment and its limits are enforced, memory by out-of-memory kills and CPU by throt­tling [1105]. A Job retries a failed pod six times by default [1106]. The doc­u­mented limits are 110 pods per node and 150,000 per clus­ter [1137], so DSec's 380,000 peak would need about 2.5 clus­ters (com­puted). Sand­man, which sets requests equal to limits at 2 vCPU and 8 GiB, reports five lessons [1104]:

1. Pace cre­ates with a token bucket. In a 10,000-request burst a default open-source con­troller deliv­ered 1,124 usable sand­boxes, and pacing plus retries all 10,000.
2. Spread place­ment: 0 failed first IP assign­ments against 254 by default.
3. Address pods directly: 0.05 s to the first com­mand against 2.3 s through Ser­vices.
4. Own cleanup after can­cel­la­tion, or retried cre­ates leave orphan pods.
5. Kill the whole process group on time­out.

DSec skips Kuber­netes: its engine picks the least loaded of $k$ sam­pled nodes, bursts to cloud VMs above 80% uti­liza­tion, and keeps roll­out state alive across pre­emptible GPU jobs [10]. rStar2-Agent runs its tools on its 64-MI300X train­ing clus­ter's CPUs [1138], shar­ing a fail­ure domain.

Den­sity and over­com­mit

In Sand­man CPU use aver­aged 10.5% of requests and memory 1.9% [1104]. In DSec about 90% of sand­boxes use at most 5% of their CPU request, and nodes ran stably with at least 3,200 con­tain­ers or 800 microVMs [10]. Kimi K3 over­com­mits memory up to 6.5x, and a paused sand­box holds no memory or CPU [1066]. Fire­cracker has run memory and CPU over­sub­scrip­tion of up to 10x in pro­duc­tion, and its authors explain why pack­ing pays: the ratio of peak to mean use falls roughly as $n^{-1/2}$ for $n$ uncor­re­lated work­loads [1087]. But an RL batch starts together, so its work­loads are cor­re­lated and that ben­e­fit is weak­est in bursts (this guide's infer­ence).

Keep infra­struc­ture out of the reward

Clas­sify every failed tra­jec­tory as model, envi­ron­ment or infra­struc­ture, and never score the last two as zero. GLM-5 excludes sam­ples that fail from envi­ron­ment col­lapse [1139], Sand­man retries them and never scores them zero [1104], and Inspect marks an error state [1103]. Fail­ures com­pound. With per-sand­box suc­cess $\rho$, a syn­chro­nous batch of $n$ is clean with prob­a­bil­ity

only ${0.997}^{512}=21.5\%$ (com­puted). One retry lifts Sand­man's lowest per-attempt avail­abil­ity, 91.16% in cam­paign C (reported), to $1-{0.0884}^2=99.2\%$ if retries fail inde­pen­dently (com­puted), con­sis­tent with its reported at least 96.69% per tra­jec­tory with one retry [1104].

Same patch, same reward

Of 500 SWE-bench Ver­i­fied ref­er­ence patches, 15 got dif­fer­ent ver­dicts across Sand­man's exe­cu­tion con­di­tions, 11 with out­bound traf­fic denied [1104]. On Ter­mi­nal-Bench 2.0 the most and least resourced setups dif­fered by 6 points (p < 0.01), and 3x resource head­room cut infra­struc­ture errors from 5.8% to 2.1% with scores within noise [1140]. Its author advises a sep­a­rate request and limit per task, unlike Sand­man's equal values, and doubt­ing gaps below 3 points until con­fig­u­ra­tions match [1140]. Pin and declare, in order [1104]:

1. Images by digest (reg­istry expiry still bites).
2. Exe­cu­tion con­di­tions, checked by an in-pod probe: net­work, CPU, memory, user, time limit, hash seed and time zone.
3. A pre­flight per task (ref­er­ence passes, empty patch fails) at 0.06⁠–⁠0.14 pod-hours, or 275⁠–⁠641 pod-hours for 4,578 tasks (com­puted).
4. Grad­ing in a fresh sand­box from the same image, because git reset and git clean do not restore every­thing.
5. Seeds and snap­shot reseed­ing (Sec­tion 14.3).

What the fleet costs next to its GPUs

Hold $k_{\text{sb}}$ sand­boxes per GPU for the whole run, priced $c_s$ per sand­box-hour at their reserved size against $c_g$ per GPU-hour:

DeepSWE and Sand­man's study runs (128 tra­jec­to­ries, 16 B200 GPUs) have $k_{\text{sb}}=8$ [9, 1104]. MiniMax-M1's rental of 534,700 USD for 512 H800s over three weeks [656] is 2.07 USD per GPU-hour, assum­ing 21 days (com­puted). At an illus­tra­tive 0.03 USD per sand­box-hour the fleet costs $8\times 0.03/2.07=11.6\%$ of the GPUs, and 1.7% against 14 USD GPUs. At 0.12 USD it is 46%. Sand­man reports 1.7⁠–⁠4.0% at on-demand list prices, depend­ing on how CPU-node cost is split [1104], a dif­fer­ent price basis from the 2.07 USD case. Its esti­mate also uses 3.30⁠–⁠3.75 sand­box-hours per GPU-hour rather than 8, and B200s at about 20 USD per GPU-hour list price [1104]. At 10,000 sand­boxes on 64 GPUs ($k_{\text{sb}}\approx 156$) the fleet costs more than the GPUs. The cost panel of Figure 76 shows the ratio.

HumanEval's har­ness sets os.system, os.fork and sim­i­lar func­tions to None before run­ning a com­ple­tion. Why do its authors still say it is not a secu­rity sand­box?

The guard is Python-level patch­ing inside the same process, user and kernel. Other mod­ules or C exten­sions reach the same func­tions, and noth­ing limits the net­work or the filesys­tem. It pro­tects the test from care­less code, not the host from hos­tile code, which is why the Codex team ran their eval­u­a­tions under gVisor with an eBPF fire­wall [1061, 699]. A real bound­ary sits below the process (Sec­tion 14.1).

You restore 16 microVMs from one snap­shot to start a GRPO group, and each gen­er­ates a UUID for its trace ID. What hap­pens, and how much does VMGenID fix?

If the UUID library's random state was in the snap­shot, all 16 draw the same "random" ID, so traces col­lide with prob­a­bil­ity 1, against about $9.4\times {10}^{-26}$ for a mil­lion fresh IDs (com­puted). VMGenID makes Linux 5.18 and later reseed the kernel's gen­er­a­tor on resume, but user-space state is still copied, so libraries must reseed or draw from the kernel after restore [1108, 1094] (Sec­tion 14.3).

SWE-bench counts a test miss­ing from the test log as a fail­ure. Which reward hacks does that rule defeat, and what else must hold for it to work?

It defeats exit­ing early and rais­ing excep­tions to skip tests, the two hacks that became sys­temic in a fron­tier RL run: a test that never ran cannot count as a pass. It works only if grad­ing hap­pens in a fresh con­tainer the agent never touched and reads per-test status, not the exit code [814, 733] (Sec­tion 14.4).

A grader has a 10% false-neg­a­tive rate and a 5% false-pos­i­tive rate, both inde­pen­dent of the policy. Does RL still opti­mize the right thing?

In expec­ta­tion, yes. Observed reward is $0.9\mu +0.05(1-\mu )$, which still rises with the true suc­cess rate $\mu$, with slope $1-0.10-0.05=0.85$. The best policy is unchanged, but the expected signal is 15% weaker and the noise adds vari­ance, so learn­ing is slower. It breaks when false pos­i­tives depend on the policy: a policy that learns which wrong answers pass raises its own false-pos­i­tive rate, and that is reward hack­ing (Sec­tion 14.6).

Your SWE agent's sand­boxes spend 90% of their lives wait­ing for the model, and the infer­ence engine wants 256 tra­jec­to­ries in gen­er­a­tion. How many sand­boxes must be live, and why do fleets over­com­mit CPU?

$n_{\text{sb}}=n_{\text{gen}}/w=256/0.9\approx 284$ (com­puted). Almost all of them are idle at any moment. Sand­man mea­sured 74⁠–⁠91% wait­ing and mean CPU use of 10.5% of requests, and DSec found 90% of sand­boxes using at most 5% of their CPU request [1104, 10]. Pack­ing by requests wastes most of the fleet, so plat­forms over­com­mit CPU and reclaim memory from paused sand­boxes, and memory becomes what binds.

15. The research engineer's playbook

A run plan joins the ear­lier chap­ters into one set of deci­sions. Choose a hypoth­e­sis, define the com­par­i­son, budget the exper­i­ment and decide how to respond when the evi­dence dis­agrees. The worked plan fol­lows 2,048 H100s for 30 days through that account­ing.

Model devel­op­ment is lab work. The Smol Train­ing Play­book calls machine learn­ing "very much an exper­i­men­tal sci­ence" [13], and Karpa­thy warns that a buggy net­work will usu­ally "train but silently work a bit worse" [943]. The costly bugs do not crash, and habits exist to catch them.

The run­ning hook is a reser­va­tion of 2,048 H100s for 30 days, the same GPU count as the clus­ter of 2,048 H800 GPUs that trained DeepSeek-V3 [2]. Sec­tions 15.1 to 15.3 cover method, loss curves and debug­ging. Sec­tion 15.4 plans the run, and Sec­tions 15.5 to 15.7 col­lect the plan­ning for­mu­las, what to study next and the open prob­lems. The run­ning exam­ples and the nota­tion are intro­duced in Sec­tion 0.3.

15.1 The experimental method

How do teams decide what goes into a run they can afford only once, and how do they know a small exper­i­ment is telling the truth? The method guards against two errors: believ­ing noise, and believ­ing a small-scale result that does not sur­vive scale.

Start from a proven base­line, as Llama 3 did from Llama 2 and Kimi K2 from DeepSeek-V3, and change one thing at a time: "Never change any­thing unless you’ve tested that it helps" [13]. Sort the hyper­pa­ram­e­ters into sci­en­tific ones (being mea­sured), nui­sance ones (re-tuned for each sci­en­tific value so the com­par­i­son is fair) and fixed ones [393]. In an opti­mizer com­par­i­son the learn­ing rate is a nui­sance hyper­pa­ram­e­ter, since one opti­mizer's best set­tings can be poor for another [418].

SmolLM3's work­horse proxy was a 1B model on 45B tokens (45 tokens per param­e­ter, against the com­pute-opti­mal 20 of Sec­tion 6.2) at 42k tokens per second per GPU on eight H100s (reported) [13], so a run takes $45\times {10}^9/(\mathrm{42,000}\times 8)=1.34\times {10}^5$ s, or 1.55 days (com­puted). A good abla­tion metric rises mono­ton­i­cally, beats random at the proxy's scale, ranks models con­sis­tently and has low seed noise. Format mat­ters: a 1.7B model beat random on mul­ti­ple-choice MMLU only after 6T tokens [13], and con­tin­u­ous scores lift deci­sion accu­racy on two code tasks "from triv­ial to 80%", though not on math [258].

When is a dif­fer­ence real?

Seed-to-seed noise in bench­mark scores is mea­sured in Sec­tion 10.9. For ten 7B models that differ only in ini­tial­iza­tion seed, with a fixed data order, Madaan et al. report seed stan­dard devi­a­tions $\sigma$, aver­aged over 21 check­points from 10B to 210B tokens, of 0.21 accu­racy points on HellaSwag, 1.11 on HumanEval and 2.15 on COPA [745]. With data order also varied, DataDecide finds seed stan­dard devi­a­tions as high as 2 points at 1B for some recipes [258].

If arms A and B each aver­age $n$ inde­pen­dent seeds with Gauss­ian noise, their dif­fer­ence has stan­dard devi­a­tion $\sigma \sqrt{2/n}$. A two-sided 95% test then calls a gap real when it exceeds the min­i­mum detectable dif­fer­ence (MDD):

For 80% power against a true gain $\Delta$, the gain must clear the MDD by another $z_{0.80}\sigma \sqrt{2/n}$, so each arm needs

Both treat $\sigma$ as known. Esti­mat­ing it from three seeds per arm with a t-test cuts the power from 83% to about 60% (com­puted by sim­u­la­tion). For $k$ com­par­isons use $z_{1-0.025/k}$, which is 3.02 at $k=20$ (Bon­fer­roni, com­puted).

In Table 15.1, MMLU's cloze form is about 2.6 times less noisy than its near-chance mul­ti­ple-choice form ($0.57/0.22$, com­puted) [745]. The last column, the item-sam­pling half-width $1.96\sqrt{\overline{s}(1-\overline{s})/N_{\text{items}}}$ (0.90 points for HellaSwag, com­puted, 0.93 reported), describes one model on the pop­u­la­tion of ques­tions, not whether retrain­ing would change a com­par­i­son. Error bars and paired com­par­isons for bench­mark scores are derived in Sec­tion 10.4.

Decide with a rule, and know what trans­fers

Adopt a change only "based on strong evi­dence" [393]. The NanoGPT speedrun requires p<0.01 that a record's mean val­i­da­tion loss reaches the 3.28 target, except for sys­tems-only changes [415]. Trans­fer is mixed. Rank­ing data recipes at 150M picks the 1B winner in about 80% of com­par­isons [258]. OLMo's lad­ders, at 1% of the target com­pute, land within 2 points on four tasks but miss by 4 on aver­age [473]. Re-exam­in­ing 46 tasks from one ear­lier study, Lourie et al. find smooth, pre­dictable down­stream scal­ing for only 18, "39% of the time" [474]. How to run small-scale abla­tions and when to trust them is Sec­tion 3.9. How to fit a scal­ing law and how far to trust its extrap­o­la­tion is Sec­tion 6.4. Com­pare after the decay, since opti­mizer rank­ings "can flip during train­ing due to learn­ing rate decay" [418].

Budget the research too, as Sec­tion 15.4 does with SmolLM3's 58% abla­tion share. The usual traps are single-seed wins below the MDD, report­ing only the best of many vari­ants (one lab tested 27 pri­vate Arena vari­ants before a release [805]), under­tuned base­lines [413] and an untested "seem­ingly inno­cent library upgrade" [13].

How to do it your­self. Mea­sure your own $\sigma$ with five or more base­line seeds that vary ini­tial­iza­tion and data order, or pool a pub­lished esti­mate. Three seeds per arm detect an effect of about $2.3\sigma$ with 80% power, and smaller effects need $n\approx 16(\sigma /\Delta {)}^2$ seeds per arm (com­puted). Com­pare after the decay, and log each deci­sion with its seeds, noise and commit.

15.2 Reading loss curves

You are watch­ing a loss plot and a gra­di­ent-norm plot at 3 a.m. Which wig­gles matter, and what does each one sug­gest? The data sets the loss level, so a mix­ture change shifts it. The learn­ing-rate sched­ule sets the slope, so a decay bends it. Sudden depar­tures come from a batch meet­ing the cur­rent param­e­ters and opti­mizer state.

At step 0 the loss should be near $\ln V$ (11.76 nats for Llama 3, com­puted) [943]. The ln V check at step 0 is in Sec­tion 12.5.

In Table 15.2 the first four rows are expected. The loss "Decreases Dra­mat­i­cally in Decay Stage" [354] and tracks the sched­ule [1142], so never com­pare a con­stant-rate run with a cosine run half­way through. Cosine, WSD and con­stant-with-cooldown sched­ules are com­pared in Sec­tion 5.2. Induc­tion heads, atten­tion heads that com­plete a sequence [A][B] … [A] with [B], form at roughly 2.5 to 5B tokens, the only non-convex stretch of the loss in the models stud­ied [424]. Level shifts follow changes in data, batch and sequence length [7, 6]. Why loss spikes happen is explained in Sec­tion 5.5, which also defines QK-norm and z-loss.

Lead­ing indi­ca­tors

The loss lags. In the run before OLMo 2 (OLMo-0424), gra­di­ent-norm spikes "often pre­ceded train­ing loss spikes" [7]. Kimi K2 caps the max­i­mum atten­tion logit at 100 with QK-Clip after seeing it pass 1,000 in an ear­lier run [72]. AdamW's update RMS, with gra­di­ent $g_t$ and second-moment esti­mate $u_t$,

should stay near 1. In Worts­man et al.'s CLIP ViT runs (vision-lan­guage, not LLM pre­train­ing) it spiked "1⁠–⁠8 iter­a­tions before a loss spike" [396]. What to mon­i­tor during a run is in Sec­tion 12.5.

A simple detec­tor (illus­tra­tive) keeps a moving mean ${\mu }_t$ and vari­ance ${\nu }_t$ of the step loss ${\ell }_t$ with weight $\alpha =0.01$, about a 199-step window ($2/\alpha -1$), and scores each step against the pre­vi­ous sta­tis­tics so a spike cannot dilute its own score:

Flag $z_t>6$ when the gra­di­ent norm is also above its 99.9th per­centile, and log the batch ids.

Then act by type. On a recov­er­able spike, con­tinue. On slow recov­ery, rewind about 100 steps and skip a few hun­dred batches [34]. PaLM's restart-and-skip fix is told in Sec­tion 5.5. On diver­gence, rewind fur­ther and lower the learn­ing rate, as OPT-175B did, where cut­ting gra­di­ent clip­ping from 1.0 to 0.3 also helped [345]. Struc­tural fixes belong to Chap­ter 5.

15.3 Debugging from symptom to cause

Some­thing looks wrong. Where in the stack do you look first? Six layers can pro­duce the same symp­tom: data, model, opti­mizer, numer­ics, par­al­lel­ism and hard­ware. So start from the symp­tom, list the layers that can pro­duce it, and order the checks by cost.

Silent bugs are cheap­est to catch before the run:

1. The step-0 loss is close to $\ln V$ [943].
2. The model can "over­fit a single batch of only a few exam­ples" [943].
3. The token ids, masks and doc­u­ment bound­aries decoded right before the model call look right [943].
4. A loss on one posi­tion gives zero gra­di­ent to later posi­tions and other batch ele­ments, which rules out mask and reshape bugs [943].
5. Your har­ness repro­duces pub­lished scores for ref­er­ence models [13].
6. The GPUs pass a stress test (SmolLM3's pre­flight found two throt­tling), and check­point saving and auto-resume work [13].
7. Seeds follow the par­al­lel­ism rules in the box below.

The sys­tems rows of Table 15.3 need a word of mech­a­nism. "Even a single strag­gler can slow down thou­sands of other GPUs" [6], and at MegaScale's size "Fail­ures and strag­glers are the norm rather than the excep­tion" [47]. NVLink faults stall ker­nels "with­out return­ing a clear error code". Llama 3 diag­nosed such hangs with its NCCLX library and PyTorch's NCCL flight recorder, which keeps col­lec­tive meta­data and stack traces in a ring buffer. Its 405B run also swung 1⁠–⁠2% in through­put with the time of day, as day­time heat throt­tled GPU clocks [6]. NaNs point to numer­ics or hard­ware: BLOOM's team hypoth­e­sizes that float16's narrow range caused the irre­versible diver­gence of its 104B pro­to­types [401], and Llama 3 sus­pects silent data cor­rup­tion among its hard­ware faults [6]. Hangs, NaNs and slow­downs have full play­books in Sec­tion 12.6, and strag­glers and silent data cor­rup­tion in Sec­tion 12.2.

Every hunt ends in bisec­tion. List what dif­fers between a known-good con­fig­u­ra­tion and the fail­ing one, then test each dif­fer­ence on the cheap­est setup that repro­duces the symp­tom. Because SmolLM3's team had derisked every­thing else, they could "pin­point TP as the only plau­si­ble cause and fix the bug within a single day", a bug they caught because "down­stream eval­u­a­tions were lag­ging behind expec­ta­tions" while the loss looked fine [13]. For insta­bil­i­ties, the tuning play­book adds a probe: run about 500 steps at twice the best learn­ing rate, "but eval­u­ate every step" [393].

How to do it your­self. Run the pre­flight list on every new con­fig­u­ra­tion and keep a known-good one to return to. When a symp­tom appears, find its row in Table 15.3, run the cheap­est check, and bisect only what dif­fers from known-good.

15.4 Planning a run: 2,048 H100s for 30 days

You have 2,048 H100s for 30 days. What do you train, how do you lay it out, and what should you expect to go wrong? A run plan is a chain of for­mu­las: budget, size, tokens, layout, fail­ures, check­points, eval­u­a­tions and a research reserve. Each link has one uncer­tain input, and the plan should show it. Step 0 is the Smol Train­ing Play­book's first ques­tion, "Should we even be train­ing this model?" Pre­train only for novel research, spe­cific pro­duc­tion needs or a gap in open models [13]. We assume the answer is yes.

Step 1: the com­pute budget

The budget mul­ti­plies five inputs:

Here ETTR, the effec­tive train­ing time ratio, is "the ratio of pro­duc­tive run­time to avail­able wall­clock time of a job run" [903], and MFU is mea­sured over pro­duc­tive time. For this plan (com­puted),

The peak is dense BF16, half the 1,979 TFLOP/s NVIDIA lists with spar­sity [4]. MFU 0.40 and ETTR 0.90 follow Llama 3's reported 38⁠–⁠43% MFU and over 90% effec­tive train­ing time [6]. That is 1,474,560 GPU-hours and one twen­ti­eth of Llama 3 405B's com­pute. MFU 0.30 or 0.55 would give $1.42$ or $2.60\times {10}^{24}$ (com­puted). MFU con­ven­tions are in Sec­tion 1.3, ETTR in Sec­tion 12.8.

Figure 77 walks the nine steps with live num­bers. Drag MFU and every output moves. Change the model and its excess-loss marker barely moves along the flat curve. Shorten the check­point write and the over­head falls.

Step 2: the com­pute-opti­mal size

Com­pute-opti­mal sizing is derived in Sec­tion 6.2. For $ℒ(N,D)=E+A/N^{\alpha }+B/D^{\beta }$ with $C=6ND$, where $\alpha$ and $\beta$ are the Chin­chilla expo­nents,

At $1.89\times {10}^{24}$ FLOPs the answer depends on the fit: 55.1B with Hoff­mann et al.'s printed con­stants, 69.4B unrounded, 125.5B at 20 tokens per param­e­ter, 132.9B with Besiroglu et al.'s refit and 147.7B with Llama 3's token fit (Table 15.4, com­puted) [40, 446, 473, 6].

Do not extrap­o­late from printed coef­fi­cients. Llama 3 fits $D^{\star }=A{C}^{\alpha }$ in its own sym­bols with $(\alpha ,A)=(0.53,0.29)$ and reports 402B param­e­ters on 16.55T tokens at $3.8\times {10}^{25}$ FLOPs [6]. The printed pair gives 10.46T tokens and 605B param­e­ters, and no pair that rounds to it reaches 14.3T (com­puted). Quote 402B and 16.55T as reported.

Step 3: the infer­ence-aware size

Sizing a model for its serv­ing load is Sec­tion 6.5. Train­ing costs $6N$ FLOPs per token and serv­ing about $2N$, so at the com­pute-opti­mal Besiroglu loss ${ℒ}^{*}=1.944$ we solve

Sar­dana et al. advise "smaller and longer than Chin­chilla-opti­mal" under heavy demand (Table 15.5) [70], as Llama 3 does for its small models [6]. Memory-bound decode makes serv­ing FLOPs dearer than the table assumes (Sec­tion 13.2). From ${10}^{13}$ served tokens on, the infer­ence-aware model needs more than the 30 days reserved, so the reser­va­tion caps how far you can trade size for tokens.

Step 4: the model and the tokens

The Llama 3 70B shape (80 layers, width 8,192, 8 key/value heads, vocab­u­lary 128,256) has 70.55B param­e­ters (com­puted) [6]. At 40% MFU count­ing $6N$ FLOPs per token, that is 935 tokens per second per GPU and 4.47T tokens in 30 days. Llama 3 70B's 15T tokens would take 101 days here (com­puted).

Step 5: the layout and the memory

The Ultra-Scale Play­book says "At 1024+ GPU scale, a rec­om­mended setup may be tensor par­al­lel­ism (TP=8) with data par­al­lel­ism (ZeRO-2) and pipe­line par­al­lel­ism." [44] With $p_{\text{tp}}=8$ and $p_{\text{pp}}=4$, 2,048 GPUs form 64 data-par­al­lel repli­cas, each a full copy of the model on 32 GPUs. A 16,777,216-token batch is 32 sequences per replica, so $m=32$ and the bubble of the one-for­ward-one-back­ward (1F1B) pipe­line sched­ule (Sec­tion 11.4) is $3/32=9.4\%$ (com­puted) [41]. The gra­di­ent all-reduce sends 8.68 GB per GPU [851], 0.17 s at 50 GB/s [137] against an 8.76 s step (com­puted).

Acti­va­tion memory is derived in Sec­tion 1.5, and here we only divide:

We use ZeRO-1 rather than the quoted ZeRO-2, as DeepSeek-V3 did with pipe­line par­al­lel­ism [44]. ZeRO-2 would only shrink the 4.11 GiB of gra­di­ents. The terms are BF16 weights and gra­di­ents, ZeRO-1 opti­mizer states (sharded across the repli­cas, Sec­tion 11.2) [22] and Kor­thikanti et al.'s acti­va­tions with sequence par­al­lel­ism and selec­tive recom­pu­ta­tion [45], a GPT-style for­mula and so illus­tra­tive here. The total is about 29.9 of 74.5 GiB (Table 15.6).

Step 6: expected fail­ures

Fail­ure rates are in Sec­tion 12.1, and we only scale them. Llama 3 reports 419 unex­pected inter­rup­tions in 54 days on up to 16K GPUs [6]. Assum­ing 16,384 GPUs through­out, 2,048 GPUs see a job mean time to fail­ure (MTTF) of 24.7 hours and about 29 inter­rup­tions in 30 days (com­puted). Meta's RSC-1 and RSC-2 rates, 6.50 and 2.34 fail­ures per 1,000 node-days [903], give $M=1/(n_{\text{nodes}}{r}_f)$ of 14.4 and 40.1 hours, or about 50 and 18 inter­rup­tions (com­puted). The sets count dif­fer­ent fail­ures, so plan for 18 to 50.

Step 7: the check­point inter­val

Check­point size and the Young inter­val are in Sec­tion 12.3. With block­ing write stall $\delta$ and restart cost $R$, Young's inter­val and Koko­lis et al.'s expected ETTR are [931, 903]

At $M=24.7$ hours, 5-minute writes give ${\tau }^{*}=122$ min­utes, 8.9% over­head with a 10-minute restart and an ETTR of 0.915, which matches Step 1's 0.90. Fif­teen-second writes give 27 min­utes and 2.5% (com­puted). The opti­mum is flat, Daly refines it for slow writes [932], and Aper­tus set its cadence this way [1143]. A full check­point of 70.55B param­e­ters at 14 bytes each (BF16 weights, FP32 master weights and two Adam moments) is 0.99 TB, so a 15-second write needs about 66 GB/s (com­puted).

Steps 8 and 9: eval­u­a­tions and the research reserve

SmolLM3 eval­u­ated every 10B tokens at about 1.5 GPU-hours for 3B param­e­ters [13]. Here that is 447 eval­u­a­tions, about 1.1% of the reser­va­tion if cost scales with size (illus­tra­tive). Eval­u­a­tions are many jobs but little com­pute, 92.9% of jobs and 0.8% of resources in one trace [905]. Keep a pre­de­ces­sor's check­points as ref­er­ences (Sec­tion 15.3).

Then budget the research. SmolLM3's abla­tions and debug­ging cost 161,280 GPU-hours against 276,480 for its main run [13]. That 58% ratio adds 860,160 GPU-hours, 17.5 days of this clus­ter (com­puted). Reused recipes cost less, since Olmo 3's 32B "recipe was devel­oped on 7B or smaller" [12]. New pre­ci­sion costs more: DeepSeek-V3 derisked FP8 at about 16B and 230B param­e­ters [2]. Report final-run and all-in GPU-hours and the wall-clock path, as Olmo 3 does: about 56 days on 1,024 H100s, crash resump­tions included [12].

The pre­flight check­list [13]:

• a GPU stress test (SmolLM3's found two throt­tling GPUs),
• stor­age that will not evict data shards,
• check­point save and auto­matic resume, tested,
• live loss, gra­di­ent norm, through­put, GPU health and eval­u­a­tions,
• a spare node with the data pre­loaded,
• expected through­put writ­ten down. SmolLM3 expected 13,500 to 14,000 tokens per second, read here as per GPU (11T tokens on 384 GPUs in 30 days aver­ages about 11,000, com­puted).

15.5 Ten formulas for planning a run

Which for­mu­las should you be able to write from memory, and where in this guide is each one derived? Start with the plan­ning chain of Sec­tion 15.4. Each card links the sec­tion that derives it.

Every other for­mula is col­lected in Sec­tion 16.1, and every term, this chap­ter's included, in Sec­tion 16.2. The ones you will reach for next are the KV cache and decode floor (Sec­tion 13.2), the col­lec­tive cost model (Sec­tion 11.1), the DPO loss (Sec­tion 8.7), the GRPO advan­tage (Sec­tion 9.3), unbi­ased pass@k (Sec­tion 10.3) and the min­i­mum detectable dif­fer­ence (Sec­tion 15.1).

15.6 Where to go next: courses, codebases and the speedrun

After this guide, what should you work through, in what order, and what does the NanoGPT speedrun teach about how improve­ments com­pound? Read­ing builds a map, but skill comes from writ­ing the code and hit­ting the bugs. Table 15.7 pairs public resources with chap­ters.

A sug­gested path (this guide's advice):

1. Repro­duce GPT-2 124M and pass the loss-at-ini­tial­iza­tion and over­fit-one-batch checks.
2. Re-derive one speedrun record's gain with seeds and a min­i­mum detectable dif­fer­ence.
3. Do CS336 assign­ments 1 to 3.
4. Repro­duce the Ultra-Scale Play­book's memory num­bers on a multi-GPU node.
5. Work the How to Scale Your Model exer­cises.
6. Read the Smol Train­ing Play­book and write your own Sec­tion 15.4 plan.
7. Run nanochat end to end.

The speedrun: how gains com­pound

The speedrun asks how fast 8 H100s can reach 3.28 val­i­da­tion loss on FineWeb, which llm.c's GPT-2 repli­ca­tion reached in 45 min­utes on the same 8 H100s (90 min­utes on 8 A100s). Records must show p < 0.01 that they hit the target, except changes that only speed up the sys­tems [415]. The record fell from 45 min­utes on 28 May 2024 to 0.665 min­utes with record 92 on 30 August 2026, using under 330M tokens against 10B [415]. With halv­ing time

that is 67.7 times faster in 27.1 months, a halv­ing every 4.45 months. Records 1 to 21 halved the time every 2.0 months, records 21 to 92 every 8.9 months, or 13.2 with­out record 92 (com­puted from the README as of Novem­ber 2026). In 2024, 16 records after the base­line gave 12.6 times. In 2025, 39 gave 1.9 times (com­puted). In Figure 78, look for the steep first eight months, the long flat stretch, and the drop at record 92, which cut 41% by bundling sam­pled soft­max, an FP8 MLP, a hashed n-gram table, a new opti­mizer stack and CUDA graphs [415]. Returns shrink on aver­age, but bun­dles can still break the trend.

Label­ing each record with one cat­e­gory (this guide's judg­ment) and sum­ming log-speedups gives archi­tec­ture 34.5%, sev­eral changes at once 26.0%, atten­tion and con­text 12.5%, opti­mizer 10.2%, sys­tems 8.7%, pre­ci­sion 3.8%, frame­work upgrades 2.9%, sched­ule and batch 1.3%, data pipe­line 0.8% and eval­u­a­tion 0.1%, with two re-tim­ings of record 21 at −0.6% (com­puted).

Do speedrun tricks trans­fer? Muon claims "~1.5x better sample-effi­ciency" there [415], and com­pet­i­tive tasks "solve the under­tuned base­line prob­lem" [413]. The README con­cedes some meth­ods "are unlikely to scale" [415]. Moon­light reports Muon match­ing AdamW with about 52% of the FLOPs [409], yet with tuned base­lines the gains shrink with scale (Sec­tion 15.1) [418]. SmolLM3 saw "more fre­quent diver­gence with Muon and AdeMAMix" at 3B and kept AdamW, though its team sus­pects a par­al­lel­ism bug [13]. And agents recover under 20% of human speedups with­out hints. The best, o3-mini, reaches about 40⁠–⁠46% with detailed hints [1150].

15.7 Open problems as of late 2026

What does nobody know how to do yet, accord­ing to the people who train these models? Six prob­lems recur in public reports as of Novem­ber 2026.

Data. Vil­lalo­bos et al. expect train­ing sets to match the stock of public human text "between 2026 and 2032" [457]. Rep­e­ti­tion buys time: four epochs cost an 8.7B model only 0.5% [209], and Sec­tion 3.6 derives what repeated data is worth. Kimi K2 calls token effi­ciency "a crit­i­cal coef­fi­cient in the scal­ing of large lan­guage models" [72]. Open: how far syn­thetic text can replace human text, and how to mea­sure data qual­ity with­out train­ing.

Eval­u­a­tion. Small-scale deci­sions rest on noisy bench­marks [745, 259], and in one re-exam­i­na­tion of 46 tasks down­stream scal­ing was pre­dictable "39% of the time" [474]. Arena scores are biased by selec­tive dis­clo­sure [805]. Open: bench­marks with high signal-to-noise at small scale that still matter at the fron­tier.

RL scal­ing. ScaleRL finds "the field lacks pre­dic­tive scal­ing method­olo­gies com­pa­ra­ble to those estab­lished for pre-train­ing", after more than 400,000 GPU-hours of study [483]. Olmo 3 writes that RL theory "is less devel­oped", and its final RL run lost at least a day to insta­bil­ity [12]. Open: what sets RL's ceil­ing, and how to derisk an RL recipe (Chap­ter 9, Chap­ter 14).

Reli­a­bil­ity. Koko­lis et al. project an MTTF of 0.23 hours at 131,072 GPUs [903], and Llama 3 saw power swings "on the order of tens of megawatts" [6]. Open: train­ing that sur­vives fail­ures with­out global restarts, and cheap detec­tion of silent data cor­rup­tion (Chap­ter 12).

Effi­ciency and limits. Fron­tier com­pute grew 4⁠–⁠5 times a year to May 2024 [101], and the amor­tized cost of fron­tier runs grew 2.4 times a year since 2016 [100]. Epoch judges $2\times {10}^{29}$ FLOP runs fea­si­ble by 2030, with power bind­ing first [103]. Data move­ment may cut uti­liza­tion above ${10}^{28}$ FLOPs [1151]. Open: larger crit­i­cal batches, lower pre­ci­sion with­out loss, and opti­mizer gains that sur­vive scale.

Automat­ing research. Agents recover a minor­ity of known speedrun gains [1150], yet leader­boards now list AI-cred­ited and autore­search entries [415, 1148]. Open: hold­ing agent claims to the stan­dards of Sec­tion 15.1.

An abla­tion shows +0.4 HellaSwag points with one seed per arm. Ship it?

No. With seed stan­dard devi­a­tion 0.21 points (Madaan et al., 7B), the one-seed min­i­mum detectable dif­fer­ence is $1.96\times 0.21\times \sqrt{2}=0.58$, so 0.4 is noise. Three seeds per arm lower it to 0.34, or use a con­tin­u­ous metric with better signal-to-noise. If this was the best of 20 vari­ants, Bon­fer­roni raises the bar to about 0.90 (com­puted).

Your step-0 loss with the Llama 3 tok­enizer reads 13.5 nats. What do you sus­pect?

A near-uni­form soft­max over 128,256 tokens gives $\ln \mathrm{128,256}=11.76$ nats. Higher means con­fi­dently wrong pre­dic­tions: an output layer ini­tial­ized too large, or a loss summed or nor­mal­ized over the wrong axis. Check those first. (13.5 is illus­tra­tive.)

The loss looks normal, but at 1T tokens your eval­u­a­tions trail a pre­de­ces­sor's check­points at the same token count. What do you test first?

The untested dif­fer­ences from the known-good setup, on the cheap­est repro­duc­tion. In SmolLM3 that was tensor par­al­lel­ism: a small model with and with­out TP showed every TP rank used the same ini­tial­iza­tion seed. Seed­ing with seed + tp_rank fixed it, and only the ref­er­ence check­points revealed it.

Check­point writes block for 5 min­utes and the job's MTTF is 24.7 hours. How often should you check­point, and what does a 15-second asyn­chro­nous write buy?

${\tau }^{*}=\sqrt{2\times 5\times \mathrm{1,485}}\approx 122$ min­utes, losing about 9% to writes, recom­pu­ta­tion and restarts. A 15-second stall gives 27 min­utes and about 2.5%. The 6.4-point gap over 720 hours is about 46 hours, two days of all 2,048 GPUs (com­puted).

Muon gave about 1.5x sample effi­ciency in the 124M speedrun. Should you expect 1.5x at 70B?

Not with­out evi­dence. Moon­light reports about 52% of AdamW's FLOPs, but tuned-base­line bench­marks show speedups fall­ing from 1.4x at 0.1B to 1.1x at 1.2B, and SmolLM3 saw more diver­gence at 3B. Run a ladder with tuned base­lines for both and com­pare at the end of the learn­ing-rate decay.

16. Appendix: cheat sheet, glossary and reading list

The for­mu­las, terms and read­ing list belong close to the work. Each entry below links back to the deriva­tion or expla­na­tion, so a remem­bered rule can be checked against its assump­tions before it goes into a plan.

The appen­dix has three parts. Sec­tion 16.1 col­lects 160 for­mu­las in 15 groups, one group per chap­ter, each with its mean­ing, an exam­ple value and a link to its home sec­tion. Sec­tion 16.2 defines 527 terms the guide uses, in alpha­bet­i­cal order. Sec­tion 16.3 lists fifty pri­mary sources, grouped by the parts of the guide. When a card is too terse, follow its link to the deriva­tion, the assump­tions and the fail­ure modes.

16.1 Cheat sheet: the key formulas by chapter

Which for­mula do I need, what does it mean, and what number does it give on a model I know? Find the chap­ter, find the card, then follow its link when you need the deriva­tion. Nota­tion is the guide's (Sec­tion 0.3). FLOPs is a count and FLOP/s a rate. GB means 10⁹ bytes and GiB means 2³⁰ bytes. Each exam­ple car­ries its label: (reported) for a value a paper or vendor states, (com­puted) for arith­metic done in this guide from cited inputs and (illus­tra­tive) for a made-up input chosen to show the scale.

Chap­ter 1. FLOPs and memory: the arith­metic of train­ing

Chap­ter 2. Hard­ware: accel­er­a­tors, inter­con­nects and clus­ters

Chap­ter 3. Data: from the raw web to train­ing tokens

Chap­ter 4. Archi­tec­ture and the pre­train­ing objec­tive

Chap­ter 5. Opti­miza­tion and train­ing dynam­ics

Chap­ter 6. Scal­ing laws: pre­dict­ing loss, sizing runs and spend­ing com­pute

Chap­ter 7. Mid­train­ing: anneal­ing, long con­text and con­tin­ued pre­train­ing

Chap­ter 8. Post-train­ing I: instruc­tion tuning, pref­er­ences and RLHF

Chap­ter 9. Post-train­ing II: rein­force­ment learn­ing for rea­son­ing and agents

Chap­ter 10. Eval­u­a­tion: mea­sur­ing what a model can do

Chap­ter 11. Dis­trib­uted train­ing: from one GPU to a hun­dred thou­sand

Chap­ter 12. Run­ning the run: reli­a­bil­ity, observ­abil­ity and ML oper­a­tions

Chap­ter 13. Infer­ence: serv­ing models fast and cheap

Chap­ter 14. Sand­boxes and envi­ron­ments: where models act

Chap­ter 15. The research engi­neer's play­book

16.2 Glossary

What does this term mean, and where is it explained? The glos­sary lists 527 terms in alpha­bet­i­cal order, and each entry ends with a link to the sec­tion that explains it.

(1-sqrt) cooldown

A decay shape that falls fast at first and slowly at the end, which beat linear decay in Hagele et al. at short decay lengths. Sec­tion 7.1

1F1B

A pipe­line sched­ule (PipeDream-Flush) that alter­nates one for­ward and one back­ward pass, cap­ping in-flight micro-batches near the pipe­line depth. Sec­tion 11.4

2:4 struc­tured spar­sity

At most two nonze­ros in every four values. Tensor cores skip the zeros for twice the through­put, and spec sheets often quote that sparse rate. Sec­tion 2.1

20%-over­head batch

The batch that needs 20% more data than the min­i­mum. One fifth of the crit­i­cal batch on McCandlish et al.'s trade-off curve. Sec­tion 5.3

6ND

Train­ing com­pute esti­mate: 6 FLOPs per param­e­ter per token (2 for­ward, 4 back­ward) times tokens. Sec­tion 1.2

ABF (adjusted base fre­quency)

Rais­ing the RoPE base and con­tin­u­ing pre­train­ing at the longer length. Sec­tion 7.4

Abla­tion

An exper­i­ment that changes one com­po­nent against a base­line to mea­sure its effect. Sec­tion 15.1

Accep­tance length ($\tau$)

Mea­sured mean number of tokens pro­duced per draft-and-verify cycle. Sec­tion 13.7

Accep­tance rate, draft length and draft cost ratio

In spec­u­la­tive decod­ing, $\alpha$ is the mean prob­a­bil­ity a drafted token is accepted, $\gamma$ the tokens drafted per iter­a­tion and $c$ the time of a draft step over a target step. Sec­tion 13.7

Accu­mu­la­tion pre­ci­sion

The pre­ci­sion of the run­ning sum inside a matmul. FP8 mat­muls need FP32 accu­mu­la­tion. Sec­tion 2.3

Acti­vated param­e­ters

In a mix­ture of experts, the param­e­ters that do work for a given token. They set FLOPs, while total param­e­ters set memory. Sec­tion 1.1

Acti­va­tion recom­pu­ta­tion

Stor­ing fewer acti­va­tions in the for­ward pass and recom­put­ing them in the back­ward pass, also called rema­te­ri­al­iza­tion or gra­di­ent check­point­ing. Full recom­pu­ta­tion reruns every layer and turns 6N into 8N hard­ware FLOPs per token. Sec­tion 1.5

Acti­va­tions

Inter­me­di­ate ten­sors saved in the for­ward pass for use in the back­ward pass. Sec­tion 1.5

Adam epsilon

A con­stant added to the root of the second moment. When gra­di­ents are smaller than epsilon it silently shrinks updates. Sec­tion 5.1

AdamW

Adam with weight decay applied directly to the weights, out­side the adap­tive divi­sion by the root of the second moment. Sec­tion 5.1

AdamW timescale

$\tau =1/(\eta \lambda )$ steps, the window over which AdamW's weights aver­age recent updates. $\tau /T$ is that window as a frac­tion of the run. Sec­tion 5.1

Agent-com­puter inter­face (ACI)

The com­mands an agent can use and the format of the feed­back it gets from the com­puter. Sec­tion 14.4

Algo­rithm and bus band­width (algbw, busbw)

Algo­rithm band­width is mes­sage size over time. Bus band­width mul­ti­plies it by a col­lec­tive-spe­cific factor, $2(p-1)/p$ for all-reduce, so it can be com­pared with the link rate. Sec­tion 11.1

ALiBi

A fixed per-head linear dis­tance penalty added to atten­tion logits. Sec­tion 4.6

All-gather

Every rank con­trib­utes a 1/p slice and every rank ends with the con­cate­na­tion. Sec­tion 11.1

All-reduce

Sum arrays across ranks and give every rank the sum. Imple­mented as a reduce-scat­ter fol­lowed by an all-gather. Sec­tion 11.1

All-to-all

Every rank sends a dif­fer­ent slice to every other rank. The expert-par­al­lel and Ulysses shuf­fle. Sec­tion 11.6

Alpha-beta model

The time of an $n$-byte mes­sage is $\alpha +n/W$: a fixed latency per mes­sage plus bytes over link band­width. Sec­tion 11.1

Anneal­ing (decay phase, cooldown)

The final phase of a run, also called the decay phase or cooldown: the learn­ing rate falls to zero or near zero and the loss drops sharply. Labs that switch to a curated data mix during it also call it mid­train­ing. Sec­tion 7.1

Anneal­ing probe

A short learn­ing-rate decay on a mix that con­tains can­di­date data, used to mea­sure what the data adds. Sec­tion 3.9

Approaches 1, 2 and 3

Chin­chilla's three esti­ma­tors: enve­lope of train­ing curves, IsoFLOP minima, para­met­ric fit. Sec­tion 6.2

Arith­metic inten­sity

FLOPs per­formed per byte moved to or from a memory level, usu­ally HBM. An oper­a­tion is memory-bound when its inten­sity is below the chip's ridge point. Sec­tion 1.7

Aspect ratio

Model width $d$ divided by the number of layers. Sec­tion 4.4

Asso­cia­tive recall

Retriev­ing a value given a key that appeared ear­lier in the con­text. Sec­tion 4.8

Asyn­chro­nous check­point

A short block­ing copy of state to host memory fol­lowed by a back­ground write to stor­age. Sec­tion 12.3

Atten­tion-FFN dis­ag­gre­ga­tion (AFD)

Run­ning atten­tion and mix­ture-of-experts FFN layers of the decode phase on sep­a­rate GPU pools. Sec­tion 13.6

Atten­tion logit growth (atten­tion entropy col­lapse)

Atten­tion scores grow­ing until the atten­tion weights become nearly one-hot. Sec­tion 4.3

Atten­tion sink

An early token that absorbs atten­tion mass a head does not need. Sec­tion 4.5

Aux­il­iary load-bal­anc­ing loss

A penalty $\alpha E{\sum }_i{f}_i{P}_i$, where $f_i$ is the share of tokens sent to expert $i$ and $P_i$ its mean router prob­a­bil­ity. Uni­form rout­ing min­i­mizes it. Sec­tion 4.7

Aux­il­iary-loss-free (bias) bal­anc­ing

Per-expert rout­ing biases nudged by observed load and used only to choose experts. Sec­tion 4.7

Base model

A pre­trained model before any post-train­ing. Sec­tion 8.1

Base­line and advan­tage

A number sub­tracted from the reward that does not depend on the sample, and the reward minus that base­line. Sec­tion 9.2

Batch invari­ance

A kernel prop­erty: each row's result is bit­wise inde­pen­dent of how many other rows share the batch and how a sequence is split. With­out it, tem­per­a­ture-0 serv­ing can answer dif­fer­ently under dif­fer­ent load. Sec­tion 12.7

Batch ramp

Start­ing with a small batch and grow­ing it as the crit­i­cal batch grows during train­ing (batch-size warmup). Sec­tion 5.3

Best-of-n

Sam­pling $n$ responses and return­ing the one with the high­est reward. It moves the policy at most $\ln n-(n-1)/n$ nats of KL from the sam­pler. Sec­tion 8.5

Best-of-N selec­tion bias

Infla­tion from pub­lish­ing the max­i­mum of sev­eral noisy mea­sure­ments of equally good vari­ants. Sec­tion 10.6

BF16

A 16-bit float with FP32's 8-bit expo­nent and 7 man­tissa bits: FP32's range at lower pre­ci­sion, the default train­ing format. Sec­tion 2.3

Bias cor­rec­tion

Divid­ing Adam's moments by $1-{\beta }^t$ to undo their zero ini­tial­iza­tion in early steps. Sec­tion 5.1

Binade

One power of two of range. A format's dynamic range is counted in binades. Sec­tion 2.3

Bits per byte (BPB)

Total neg­a­tive log-like­li­hood in bits divided by the UTF-8 bytes of the eval­u­ated text, $\frac{n_{\text{tok}}}{n_{\text{byte}}}·\frac{ℒ}{\ln 2}$. Unlike loss per token or per­plex­ity, it is com­pa­ra­ble across tok­eniz­ers. Sec­tion 3.8

Bit­wise deter­min­ism

Iden­ti­cal bits on every rerun from the same check­point and con­fig­u­ra­tion. Sec­tion 12.7

Bloom filter

A bit array with sev­eral hash func­tions that answers 'seen before?' with no false neg­a­tives and a tun­able false-pos­i­tive rate. Sec­tion 3.3

Boil­er­plate

Menus, foot­ers, cookie notices and ads around a page's main con­tent. Sec­tion 3.1

Boot­strap

Refit­ting on resam­pled runs to esti­mate the uncer­tainty of a fitted quan­tity. Sec­tion 6.4

Bradley-Terry model

The paired-com­par­i­son model $P(i\succ j)=\sigma (s_i-s_j)$ behind reward-model train­ing and arena rank­ings. Sec­tion 8.4

Break-even tokens per replica

$F/W$, the tokens each data-par­al­lel replica must process per step for its com­mu­ni­ca­tion to hide behind com­pute. Sec­tion 11.2

Bubble

Idle device time in a pipe­line step: $(p_{\text{pp}}-1)/(mv)$ of the ideal time for GPipe, 1F1B and inter­leaved sched­ules. Sec­tion 11.4

Budget forc­ing

Con­trol­ling think­ing length at decode time by forc­ing or sup­press­ing the end-of-think­ing delim­iter. Sec­tion 9.7

Burn-in

Stress test­ing at clus­ter bring-up to flush out weak com­po­nents. Sec­tion 12.4

Byte-level BPE

BPE whose base alpha­bet is the 256 byte values, so any string can be encoded. Sec­tion 3.7

Byte-pair encod­ing (BPE)

Build­ing a vocab­u­lary by repeat­edly merg­ing the most fre­quent adja­cent pair of sym­bols. Sec­tion 3.7

Cache salt

A per-tenant value mixed into block hashes so ten­ants cannot share, or probe, each other's cached pre­fixes. Sec­tion 13.4

Caching allo­ca­tor

PyTorch's GPU allo­ca­tor, which keeps freed memory reserved for reuse, so nvidia-smi shows it as used. Sec­tion 1.5

Canary string

A unique marker placed in bench­mark files so that corpus builders can filter them out and audi­tors can detect their use. Sec­tion 10.5

Capa­bil­ity

One slice of root's priv­i­leges, such as CAP_SYS_ADMIN, that a process can hold or drop. Sec­tion 14.1

Capa­bil­ity thresh­old

A frame­work's capa­bil­ity level that requires stronger safe­guards: High and Crit­i­cal (OpenAI), Capa­bil­ity Thresh­olds and AI Safety Levels (Anthropic), Crit­i­cal Capa­bil­ity Levels (Google DeepMind). Sec­tion 10.8

Capac­ity factor and token drop­ping

Expert slots per step as a mul­ti­ple of the even share. Tokens beyond capac­ity skip the expert. Sec­tion 4.7

Capac­ity gap

The find­ing that a teacher too strong rel­a­tive to the stu­dent even­tu­ally yields a worse stu­dent. Sec­tion 7.7

Cat­a­strophic for­get­ting

Loss of ear­lier capa­bil­i­ties when train­ing on a new dis­tri­bu­tion. Sec­tion 7.6

Causal atten­tion count

Count­ing only the lower tri­an­gle of the atten­tion matrix, half of the full count. Sec­tion 1.2

Causal lan­guage mod­el­ing

Pre­dict­ing each token from all pre­vi­ous tokens under a causal atten­tion mask. Sec­tion 4.1

cgroup v2

The uni­fied kernel hier­ar­chy that accounts for and limits CPU, memory, I/O and process counts for a group of pro­cesses. Sec­tion 14.1

Chain of thought

The rea­son­ing text a model writes before its final answer. Rea­son­ing models such as DeepSeek-R1 learn to make it long and useful through RL on ver­i­fi­able rewards. Sec­tion 9.4

Char­ac­ter train­ing

Post-train­ing for stable traits and manner, not only harm avoid­ance. Sec­tion 8.9

Chat tem­plate

The exact token format (roles, head­ers, end-of-turn tokens) that turns a list of mes­sages into one sequence. A chat model must be trained and served with the same one. Sec­tion 8.2

Check­point

A per­sis­tent snap­shot of weights, opti­mizer state, data-loader posi­tion, RNG states, step counter and sched­uler state, enough to resume exactly. Sec­tion 12.3

Check­point aver­ag­ing

Aver­ag­ing weights across steps (Polyak, EMA) or across runs from one start (soups). Aver­ag­ing spaced check­points approx­i­mates learn­ing-rate decay. Sec­tion 5.7

Chip-sec­onds per token

Pope et al.'s cost metric: number of chips times time, divided by tokens. It is pro­por­tional to cost. Sec­tion 13.3

Chosen and rejected

The pre­ferred and dis­pre­ferred responses of a com­par­i­son. Sec­tion 8.4

Chun­ked pre­fill, token budget

Pro­cess­ing a prompt in chunks over sev­eral iter­a­tions, with a cap on tokens per iter­a­tion that keeps each iter­a­tion within the TBT target. Sec­tion 13.5

CISPO

Clipped IS-weight Policy Opti­miza­tion: REINFORCE with a stop-gra­di­ent impor­tance weight capped above, so no token is ever dropped. Sec­tion 9.5

Clip-higher

Using a larger upper clip range than lower one so rare tokens can grow. Sec­tion 9.5

Cloze for­mu­la­tion (CF)

Scor­ing each answer option as a con­tin­u­a­tion of the ques­tion and pick­ing the most prob­a­ble. Sec­tion 10.2

Cold start (rea­son­ing RL)

A small super­vised set of read­able long chain-of-thought exam­ples trained before rea­son­ing RL, as in DeepSeek-R1. Sec­tion 9.4

Cold start (serv­ing)

Time from no run­ning replica to the first served token, dom­i­nated by load­ing weights and warm­ing ker­nels. Sec­tion 13.5

Col­lec­tive

A com­mu­ni­ca­tion oper­a­tion that every rank in a group calls together, such as all-reduce, all-gather, reduce-scat­ter, broad­cast or all-to-all. Sec­tion 11.1

Col­lec­tive time­out

The watch­dog time after which a rank aborts a col­lec­tive that has not com­pleted. It is a symp­tom, not a cause. Sec­tion 12.6

Colo­cated and dis­ag­gre­gated

Train­ing and gen­er­a­tion shar­ing the same GPUs in turn, or run­ning on sep­a­rate GPU pools. Sec­tion 9.8

Common Crawl

A non­profit web archive that has released a crawl of a few hun­dred TiB every month or two since 2007. Sec­tion 3.1

CompleteP

A depth exten­sion of μP that scales each resid­ual branch by $L_{\text{base}}/L$, with match­ing learn­ing-rate, weight-decay and epsilon rules. Sec­tion 5.4

Com­pute-effi­cient fron­tier

The curves $N_{\text{opt}}(C)$ and $D_{\text{opt}}(C)$: the best model size and token count for each com­pute budget. Sec­tion 6.1

Com­pute-equiv­a­lent gain (CEG)

The train­ing-com­pute mul­ti­plier that would buy the same improve­ment as a post-train­ing enhance­ment such as a scaf­fold. Sec­tion 10.7

Com­pute-, memory-, over­head-bound

Whether math, data move­ment or every­thing else (Python, dis­patch, kernel launches) sets an oper­a­tion's run­time. Sec­tion 1.7

Com­pute-opti­mal (Chin­chilla-opti­mal)

The param­e­ter count and token count that min­i­mize pre­train­ing loss for a fixed train­ing com­pute, with unlim­ited unique data. Sec­tion 6.2

Com­pute-opti­mal pre­ci­sion

The train­ing bit width that min­i­mizes loss at fixed com­pute when cost grows with bits. Sec­tion 6.6

Com­pute unit (CU), XCD

AMD's equiv­a­lent of an SM, and the die that holds 38 of them on MI300X. Sec­tion 2.8

Con­sti­tu­tional AI

Self-cri­tique and AI pref­er­ence labels guided by a writ­ten list of prin­ci­ples. Sec­tion 8.4

Con­strained decod­ing

Mask­ing tokens that would leave a gram­mar or schema, so output stays a valid prefix at every step. Sec­tion 13.9

Con­tam­i­na­tion and exploita­tion

Test items or close vari­ants in train­ing data, and con­tam­i­na­tion that actu­ally raises the score. Sec­tion 10.5

Con­text exten­sion

A late train­ing stage that raises the trained con­text length, for exam­ple from 8K to 128K tokens, by adjust­ing RoPE and train­ing on long doc­u­ments. Sec­tion 7.5

Con­text par­al­lel­ism (CP)

Split one sequence across GPUs through atten­tion as well, exchang­ing keys and values (ring or all-gather) or re-shard­ing by heads (Ulysses). Sec­tion 11.5

Con­tin­ued pre­train­ing (CPT)

Fur­ther next-token train­ing of a pre­trained model on new or reweighted data at pre­train­ing-like scale. Sec­tion 7.6

Con­tin­u­ous (iter­a­tion-level, in-flight) batch­ing

Re-form­ing the batch at every for­ward pass, so fin­ished requests leave and wait­ing ones join imme­di­ately. Sec­tion 13.4

Cooldown

Another name for the decay phase, see Anneal­ing. Sched­ules that end in one are com­pared in Sec­tion 5.2

Coor­di­nate check

Plot­ting each layer's acti­va­tion size after a few steps against width. Flat lines mean a param­e­ter­i­za­tion is imple­mented cor­rectly. Sec­tion 5.4

Copy-on-write

Shar­ing KV blocks between sequences until one writes to a block, which is then copied. Sec­tion 13.4

Core (DCLM)

DataComp-LM's cen­tered accu­racy aver­aged over 22 low-vari­ance tasks, 0 for chance and 1 for per­fect. Sec­tion 3.2

Cosine sched­ule

The learn­ing rate fol­lows half a cosine from its peak to a floor, often 10% of peak, over the planned run. Sec­tion 5.2

CoT mon­i­tor

A model that reads another model's chain of thought to flag mis­be­hav­ior. Sec­tion 9.9

Cou­pled versus inde­pen­dent weight decay

Cou­pled (PyTorch) decay shrinks weights by ${\eta }_t\lambda$ per step. Inde­pen­dent decay's shrink is not mul­ti­plied by the learn­ing rate. Sec­tion 5.1

Cov­er­age

The frac­tion of prob­lems solved by at least one of $k$ sam­ples. It equals the suc­cess rate only with a per­fect ver­i­fier. Sec­tion 6.8

Crit­i­cal batch

The batch that needs twice the min­i­mum steps and twice the min­i­mum data to reach a target loss. Below it, larger batches save steps almost for free. Sec­tion 5.3

Cross-entropy (nats per token)

The aver­age neg­a­tive nat­u­ral-log prob­a­bil­ity the model assigns to the true next token. Sec­tion 4.1

CUDA graph

A recorded sequence of kernel launches replayed with a single launch to remove CPU over­head. Sec­tion 2.4

Dan­ger­ous-capa­bil­ity and align­ment eval­u­a­tions

Eval­u­a­tions of whether a model can do seri­ous harm, and of whether it would. Sec­tion 10.8

Data abla­tion

Iden­ti­cal models that differ only in their train­ing data, com­pared on bench­marks. Sec­tion 3.9

Data-con­strained regime

Train­ing whose token budget exceeds the unique data avail­able, so data repeats. Sec­tion 3.6

Data mix­ture

The sam­pling dis­tri­bu­tion over data domains, chosen inde­pen­dently of how large each domain is. Sec­tion 3.5

Data par­al­lel­ism (DP), DDP

Repli­cate the model, split the batch, aver­age the gra­di­ents. DDP is PyTorch's imple­men­ta­tion with buck­eted all-reduce over­lapped with the back­ward pass. Sec­tion 11.2

Data wall

The point at which train­ing sets match the stock of public human text. Sec­tion 6.6

Decay to zero (D2Z)

A sched­ule whose learn­ing rate ends at zero, usu­ally by linear decay. Sec­tion 5.2

Deci­sion accu­racy

The share of pair­wise choices, such as two data recipes, whose small-scale order­ing matches their order at the target scale. Sec­tion 3.9

Deci­sion log

The writ­ten record of each exper­i­ment's hypoth­e­sis, setup, result, noise esti­mate and deci­sion. Sec­tion 15.1

Decode

The sequen­tial loop that pro­duces one token per sequence per for­ward pass, each step read­ing all weights and the sequence's KV cache. Sec­tion 13.1

Decode floor

The lower bound on a decode step: weight bytes plus the batch's KV bytes, divided by aggre­gate HBM band­width. Sec­tion 13.2

Decoder-only trans­former

An embed­ding table, a stack of iden­ti­cal blocks (causal self-atten­tion, then an MLP, each adding to a resid­ual stream) and an output head, trained to pre­dict the next token. Every run­ning exam­ple in this guide is one. Sec­tion 4.2

Decon­tam­i­na­tion

Remov­ing train­ing exam­ples that over­lap eval­u­a­tion items, for exam­ple by 8-gram match­ing on prompts. Sec­tion 8.2

Decou­pled RoPE

MLA's small sep­a­rate rotary query and key path beside the absorbed con­tent path. Sec­tion 4.5

Dedu­pli­ca­tion

Remov­ing exact and near-dupli­cate doc­u­ments or pas­sages from a corpus, with hashes, Bloom fil­ters, MinHash with LSH, or suffix arrays. Sec­tion 3.3

Delib­er­a­tive align­ment

Train­ing a rea­son­ing model to recall and reason over a writ­ten safety spec­i­fi­ca­tion, rewarded by a judge that does not see the chain of thought. Sec­tion 9.9

Delta rule

A state update that writes the cor­rec­tion between a key's target value and its cur­rent read-out. Sec­tion 4.8

Dense versus sparse peak

Spec sheets often quote 2:4 struc­tured-spar­sity through­put, twice the dense figure that train­ing and MFU use: 1,979 against 989.5 TFLOP/s for an H100 in BF16. Sec­tion 2.1

Derisk­ing

Test­ing a change at small scale until it is shown to help, or to be neu­tral within a stated tol­er­ance, before it enters the main run. Sec­tion 15.1

Design effect

Vari­ance infla­tion from cor­re­lated ques­tions in clus­ters of size $g$: $1+(g-1){\rho }_{\text{ICC}}$. Sec­tion 10.4

Devel­op­ment and held-out eval­u­a­tions

Bench­marks used for train­ing deci­sions, and bench­marks reserved to mea­sure how much those deci­sions over­fit. Sec­tion 10.9

Device mesh

The grid of ranks with one dimen­sion per par­al­lel axis, which decides which GPUs share each group. Sec­tion 11.7

DiLoCo

Low-com­mu­ni­ca­tion train­ing in which repli­cas take $H$ inner AdamW steps, then aver­age their param­e­ter change (the outer gra­di­ent) and apply an outer Nes­terov-momen­tum step. Sec­tion 11.8

Direct pref­er­ence opti­miza­tion (DPO)

A logis­tic loss on log-prob­a­bil­ity ratios that reaches the KL-reg­u­lar­ized RLHF opti­mum from fixed pref­er­ence pairs. Sec­tion 8.7

Doc­u­ment mask­ing

Block­ing atten­tion between dif­fer­ent doc­u­ments packed into one train­ing sequence. Sec­tion 7.5

Domain upsam­pling

Rais­ing the share of small, high-qual­ity domain datasets at the end of train­ing. Sec­tion 7.2

Double binary tree

Two com­ple­men­tary binary trees that each carry half the data, giving log­a­rith­mic latency at near-ring band­width (NCCL 2.4). Sec­tion 11.1

Double descent

Test error that peaks near the inter­po­la­tion thresh­old, where a model can just fit its train­ing set, before fall­ing again. Sec­tion 5.8

Dr. GRPO

GRPO with­out the per-response length nor­mal­iza­tion and the std divi­sion, which removes its length and dif­fi­culty biases. Sec­tion 9.3

Draft head

A small trained module on the target's hidden states that pro­poses tokens (Medusa, EAGLE). Sec­tion 13.7

Drop­less MoE

Expert com­pu­ta­tion with no capac­ity limit, using block-sparse or grouped matrix mul­ti­plies. Sec­tion 4.7

DualPipe

DeepSeek-V3's bidi­rec­tional pipe­line that pairs for­ward and back­ward chunks so one's com­mu­ni­ca­tion hides behind the other's com­pute, at the cost of two copies of the weights. Sec­tion 11.4

Dynamic sam­pling

Over-sam­pling prompts and keep­ing only groups with mixed rewards. Sec­tion 9.5

E4M3 and E5M2

The two FP8 encod­ings: 4 expo­nent and 3 man­tissa bits (max­i­mum 448) for pre­ci­sion, or 5 and 2 (max­i­mum 57,344) for range. Sec­tion 2.3

ECMP, hash polar­iza­tion

Equal-cost multi-path rout­ing that hashes each flow onto one path, and the cor­re­lated hash­ing across tiers that ampli­fies imbal­ance. Sec­tion 2.6

Edge of sta­bil­ity

Train­ing in which the top Hes­sian eigen­value rises to and hovers at the sta­bil­ity limit $2/\eta$ ($38/\eta$ for Adam with ${\beta }_1=0.9$) while the loss keeps fall­ing over long timescales. Sec­tion 5.8

Effec­tive com­pute

Phys­i­cal com­pute mul­ti­plied by algo­rith­mic effi­ciency gains. Sec­tion 6.9

Effec­tive con­text length

The long­est length at which a model still passes a stated qual­ity thresh­old on a long-con­text bench­mark, as opposed to the claimed window. Sec­tion 7.5

Effec­tive data

The number of fresh tokens a multi-epoch run is worth. With Muen­nighoff et al.'s decay con­stant $R_D^{\star }\approx 15.4$, four epochs are worth about 93% of fresh data. Sec­tion 3.6

Effec­tive param­e­ter count

A dense-equiv­a­lent size for a routed or low-pre­ci­sion model. Sec­tion 6.6

Effec­tive sample size

$({\sum }_i{w}_i{)}^2/(N{\sum }_i{w}_i^2)$ for impor­tance weights $w_i$: 1 on-policy, near 0 when a few weights dom­i­nate. Sec­tion 9.6

Effec­tive train­ing time ratio (ETTR)

Pro­duc­tive train­ing time divided by the wall-clock time of a job run. Meta's def­i­ni­tion includes time queued while the job is eli­gi­ble to run. Sec­tion 12.8

Elas­tic train­ing

Con­tin­u­ing on fewer (or more) resources after a fail­ure instead of wait­ing for replace­ments. Sec­tion 12.4

Elic­i­ta­tion and sand­bag­ging

Effort to draw out a model's full capa­bil­ity, and strate­gic under­per­for­mance on an eval­u­a­tion. Sec­tion 10.8

Elo scale

Bradley-Terry strengths rescaled to 400 points per factor of ten in odds: a 10-point gap is a 51.4% expected win rate. Sec­tion 10.6

Embed­ding

The learned vector of width $d$ for each token id, looked up in a $V\times d$ table at the input. The output head maps the last hidden state back to $V$ logits. Sec­tion 1.1

Emer­gent abil­ity

Wei et al.: absent in smaller models, present in larger ones. Du et al.: absent above a pre­train­ing-loss thresh­old, present below it. Sec­tion 6.7

Entropy col­lapse (RL)

Token entropy fall­ing early and stay­ing low, so answers stop vary­ing and learn­ing stalls. Not the atten­tion fail­ure of the same name, see Atten­tion logit growth. Sec­tion 9.6

Envi­ron­ment

The world side of the RL loop: reset() and step(action), which returns obser­va­tion, reward, ter­mi­nated, trun­cated and info. Sec­tion 14.5

Envi­ron­ment-token mask­ing

Giving no policy-gra­di­ent loss to tool out­puts, retrieved text and other envi­ron­ment tokens. Sec­tion 9.9

Epochs per domain

Passes over a domain implied by a mix­ture: its token share times the budget divided by its unique tokens. Sec­tion 3.5

Exe­cu­tion con­di­tions

The declared net­work access, CPU, memory, user, time limit, hash seed and time zone of a grad­ing run. Sec­tion 14.8

Expert par­al­lel­ism (EP)

Plac­ing dif­fer­ent experts of a mix­ture-of-experts layer on dif­fer­ent GPUs and moving tokens to them and back with two all-to-alls, dis­patch and com­bine. Sec­tion 11.6

Exten­sion factor (RoPE)

Target con­text length divided by trained con­text length, $k_{\text{ext}}=s_{\text{target}}/s_{\text{train}}$. Papers write $s$, which this guide reserves for sequence length. Sec­tion 7.4

Extrap­o­la­tion ratio

Target com­pute divided by the largest com­pute in the ladder. Sec­tion 6.4

FAIL_TO_PASS / PASS_TO_PASS

Tests that the ref­er­ence fix turns from fail­ing to pass­ing, and tests that pass both before and after it. Sec­tion 14.4

Fail­ure rate

$r_f$: fail­ures per node-day, usu­ally quoted per 1,000 node-days. Sec­tion 12.1

Fat tree (folded Clos), radix

A multi-tier switch net­work with equal band­width up and down at each tier. Radix is the number of ports per switch. Sec­tion 2.6

Fault-tol­er­ant HSDP

Hybrid sharded data par­al­lel­ism in which each data-par­al­lel replica is its own fault domain, so a failed replica drops out and rejoins with­out a global restart. Sec­tion 12.4

Fill-in-the-middle (FIM)

Reorder­ing a doc­u­ment into prefix, suffix, then middle so the model learns to infill. Sec­tion 3.5

Fine-grained experts and gran­u­lar­ity

Many small experts instead of a few large ones. Gran­u­lar­ity com­pares dense and expert widths, with con­ven­tions that differ by paper. Sec­tion 4.7

Fine-grained scal­ing

Giving each small tile or block of a tensor its own FP8 scale, com­puted online, so one out­lier cannot crush the rest. Sec­tion 5.6

Flaky test

A test whose out­come changes between runs on the same code. Sec­tion 14.6

FlashAttention, online soft­max

A tiled atten­tion kernel that never writes the $s\times s$ score matrix to HBM, using a soft­max com­puted block by block with a run­ning max­i­mum and sum. Sec­tion 2.4

Flight recorder

PyTorch's per-rank ring buffer of recent col­lec­tives (sequence num­bers, sizes, states, stacks), dumped on time­out. Sec­tion 12.6

Flips

Answers that change between cor­rect and incor­rect when a model is com­pressed, even if aver­age accu­racy is unchanged. Sec­tion 13.8

FLOP / FLOPs

One float­ing-point add or mul­ti­ply. FLOPs (plural) is a count. Sec­tion 1.2

FLOP/s

FLOPs per second, a rate. TFLOP/s is ${10}^{12}$ FLOP/s and PFLOP/s ${10}^{15}$. Sec­tion 1.3

Format spread

Max­i­mum minus min­i­mum accu­racy over mean­ing-pre­serv­ing prompt for­mats. Sec­tion 10.2

For­ward area S1 and anneal­ing area S2

The two terms of Tissue et al.'s anneal­ing law: the sum of learn­ing rates $S_1$, and the momen­tum-weighted sum of learn­ing-rate decreases $S_2$. Sec­tion 7.1

Foun­da­tion model

A large model pre­trained on broad data that later stages adapt to many tasks. This guide treats decoder-only lan­guage models. Sec­tion 0.1

FP32 pro­mo­tion

Peri­od­i­cally moving low-pre­ci­sion par­tial sums of a long dot prod­uct into FP32 reg­is­ters to stop accu­mu­la­tion error from grow­ing. Sec­tion 5.6

FP8

An 8-bit float­ing-point format, E4M3 or E5M2, used for mat­muls in train­ing and serv­ing with FP32 accu­mu­la­tion and fine-grained scales. Sec­tion 2.3

Frac­tion of speedup recov­ered

An agent's share of a human record's speedup when it tries to repro­duce that record. Sec­tion 15.6

FSDP and FSDP2

PyTorch's fully sharded data par­al­lel­ism, an imple­men­ta­tion of ZeRO-3: gather each unit's param­e­ters just in time and reduce-scat­ter its gra­di­ents. FSDP2 shards each param­e­ter as a DTensor. Sec­tion 11.2

Gang sched­ul­ing

Start­ing all tasks of a job together or none of them (Slurm's page uses the term for times­lic­ing). Sec­tion 12.8

Gen­er­al­ized advan­tage esti­ma­tion (GAE)

An advan­tage esti­mate that sums a learned value model's tem­po­ral-dif­fer­ence errors with weights $(\gamma \lambda {)}^l$. With $\gamma =\lambda =1$ it reduces to the return minus the value. Sec­tion 8.5

Gen­er­a­tion stall

A pause in a decod­ing request's token stream caused by another request's pre­fill sched­uled ahead of it. Sec­tion 13.5

Global batch and micro-batch

The global batch $B$ is the tokens in one opti­mizer step across all repli­cas. A micro-batch of $b$ sequences is the slice one replica or pipe­line stage pro­cesses at a time. Sec­tion 0.3

Good­hart's law

When a mea­sure becomes a target, it ceases to be a good mea­sure. Sec­tion 10.9

Good­put (serv­ing)

The high­est request rate per GPU at which a target frac­tion of requests (for exam­ple 90%) meet their TTFT and TPOT objec­tives. Sec­tion 13.3

GPU-hours

Number of accel­er­a­tors times wall-clock hours. Sec­tion 1.3

Gra­di­ent accu­mu­la­tion

Sum gra­di­ents over sev­eral micro-batches before one opti­mizer step, com­mu­ni­cat­ing once. Sec­tion 11.2

Gra­di­ent noise scale

$tr(\Sigma )/{\left|G\right|}^2$, the batch size at which gra­di­ent noise and gra­di­ent signal are com­pa­ra­ble. Sec­tion 5.3

Gray node

A machine that passes hard-fail­ure checks but runs slow on real work­loads. Sec­tion 12.4

Grokking

Gen­er­al­iza­tion that appears long after the train­ing set has been fit, seen on small algo­rith­mic datasets. Sec­tion 5.8

Group and group size

The $G$ answers sam­pled for one prompt, which share a base­line. Sec­tion 9.3

Group (block) quan­ti­za­tion

One scale per block of con­sec­u­tive values. MXFP4 uses 32-ele­ment blocks with a power-of-two scale, NVFP4 16-ele­ment blocks with an FP8 scale. Sec­tion 13.8

Grouped-query atten­tion (GQA)

Groups of query heads share one key-value head ($h_{kv}<h$). It shrinks the KV cache by $h/h_{kv}$ with­out chang­ing atten­tion FLOPs: 8x for Llama 3 70B. Sec­tion 4.5

GRPO

Group Rel­a­tive Policy Opti­miza­tion: advan­tage = the answer's z-score within its group, per-response token mean, k3 KL term in the loss. Sec­tion 9.3

GSM*

OLMo 2's 200-ques­tion devel­op­ment subset of GSM8K, used for mix deci­sions so the rest of the test set stays unseen. Sec­tion 7.3

GSPO

Group Sequence Policy Opti­miza­tion: a clipped, length-nor­mal­ized ratio for the whole response. Sec­tion 9.5

gVisor (Sentry, Gofer)

A user-space appli­ca­tion kernel writ­ten in Go: the Sentry imple­ments system calls, the Gofer opens files on its behalf. Sec­tion 14.2

Hack-ver­i­fi­able envi­ron­ment

A task with planted, detectable exploits so that hack­ing can be mea­sured exactly. Sec­tion 14.6

Hang

A job that makes no progress and reports no error until a time­out or watch­dog fires. Sec­tion 12.6

Hard­ware FLOPs uti­liza­tion (HFU)

Exe­cuted FLOP/s, includ­ing recom­pu­ta­tion, divided by peak. Sec­tion 1.3

Har­ness

The code that builds a task's envi­ron­ment, runs the agent or applies its output, runs the grader and records results. Sec­tion 14.4

HBM (high-band­width memory)

DRAM stacked beside the com­pute die. H100: 80 GB at 3.35 TB/s. Sec­tion 2.2

Health check

A peri­odic node test whose fail­ure drains the node from sched­ul­ing. Sec­tion 12.4

Heuris­tic filter

A thresh­old on a simple doc­u­ment statis­tic such as length, punc­tu­a­tion, symbol ratio or stop-word count. Sec­tion 3.2

Hier­ar­chi­cal col­lec­tive

A col­lec­tive done inside each node over NVLink first and across nodes second, so each net­work card car­ries only its GPU's share. Sec­tion 11.1

Hot spare

A pre-val­i­dated idle node kept ready to replace a failed one. Sec­tion 12.4

HSDP

Hybrid sharded data par­al­lel­ism: shard within a group of GPUs and repli­cate across groups. Sec­tion 11.2

Huber loss

A robust regres­sion loss, qua­dratic below a thresh­old $\delta$ and linear above. Sec­tion 6.4

Hybrid model

A stack mixing a few atten­tion layers with many recur­rent or SSM layers. Sec­tion 4.8

ICI, OCS

A TPU's direct inter-chip links, and the opti­cal cir­cuit switches (MEMS mir­rors) that join TPU cubes into pods. Sec­tion 2.6

Image fanout

The number of sand­boxes cre­ated from one image. Sec­tion 14.7

Implicit reward

$\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$, the reward for which a DPO-trained policy is opti­mal. Sec­tion 8.7

In-memory check­point

A copy of model state held in other machines' memory, so recov­ery skips remote stor­age. Sec­tion 12.3

Induc­tion head

An atten­tion cir­cuit that com­pletes the pat­tern [A][B] … [A] → [B]. Its for­ma­tion shows as a bump in the early loss curve. Sec­tion 5.8

Infer­ence-aware sizing

Choos­ing model size and train­ing tokens to min­i­mize train­ing plus serv­ing cost at a target qual­ity, which favors smaller models trained longer. Sec­tion 6.5

InfiniBand, RoCE v2

A loss­less, credit-based RDMA fabric, and RDMA car­ried in UDP/IP pack­ets over Eth­er­net. Sec­tion 2.6

Inoc­u­la­tion prompt­ing

Fram­ing hack­ing as accept­able in the RL prompt so that hack­ing does not gen­er­al­ize into broader mis­align­ment. Sec­tion 9.9

Inter­leaved local/global atten­tion

A stack that mixes slid­ing-window layers and full-atten­tion layers at a fixed ratio. Sec­tion 4.5

Inter­leaved sched­ule

Each device holds v non-con­tigu­ous chunks of layers, divid­ing the bubble by v at v times the point-to-point traf­fic. Sec­tion 11.4

IO com­plex­ity

The number of HBM accesses an algo­rithm needs. Sec­tion 2.4

Irre­duc­ible loss (E)

The fitted floor of the loss, read as the entropy of the data under the tok­enizer. Sec­tion 6.1

IsoFLOP pro­file

Loss against model size for runs that all use the same com­pute. Its min­i­mum is the com­pute-opti­mal size. Sec­tion 6.2

Jac­card sim­i­lar­ity

The size of the inter­sec­tion of two sets divided by the size of their union. Sec­tion 3.3

Jailer

Fire­cracker's wrap­per that con­fines the VMM process with chroot, names­paces, dropped priv­i­leges and a sec­comp pro­file. Sec­tion 14.2

Kernel fusion

Com­bin­ing sev­eral oper­a­tions into one kernel so the data is read from HBM once (also called oper­a­tor fusion). Sec­tion 2.4

KL coef­fi­cient

The weight β of the penalty that keeps the policy near the ref­er­ence policy. In DPO it is the tem­per­a­ture of the implicit clas­si­fier. Sec­tion 8.5

KL diver­gence

$D_{\text{KL}}(\pi \Vert {\pi }_{\text{ref}})={𝔼}_{y~\pi }[\log \pi (y)/{\pi }_{\text{ref}}(y)]$, in nats: how far a policy has moved from its ref­er­ence. RLHF penal­izes it, weighted by the KL coef­fi­cient. Sec­tion 8.5

KL esti­ma­tors k1, k2, k3

Schul­man's single-sample esti­mates of KL diver­gence. As losses, k2 gives the reverse-KL gra­di­ent and k3 the for­ward-KL gra­di­ent. Sec­tion 9.6

Knowl­edge dis­til­la­tion

Train­ing a stu­dent to match a teacher's output dis­tri­bu­tion (soft tar­gets) instead of, or as well as, the observed next token. Sec­tion 7.7

KV cache

The stored keys and values of all ear­lier tokens in every atten­tion layer, re-read at each decode step instead of recom­puted. A token costs $2n_{\text{layers}}{h}_{kv}{d}_h$ times the bytes per ele­ment: 128 KiB for Llama 3 8B in BF16. Sec­tion 13.2

KV-cache quan­ti­za­tion and evic­tion

Stor­ing the cache in fewer bits, or drop­ping tokens judged unim­por­tant (heavy hit­ters and atten­tion sinks are kept). Sec­tion 13.8

Label error

A bench­mark item whose key is wrong, ambigu­ous, or has sev­eral cor­rect answers. Sec­tion 10.1

Lan­guage iden­ti­fi­ca­tion (LID)

A clas­si­fier that pre­dicts each doc­u­ment's lan­guage, applied with a prob­a­bil­ity thresh­old. Sec­tion 3.1

Latency wall

A cap on train­ing com­pute set by the min­i­mum time per opti­mizer step and the crit­i­cal batch size. Sec­tion 6.9

Lazy (on-demand) image load­ing

Fetch­ing image blocks or files when first read instead of pulling the whole image before start. Sec­tion 14.7

Learn­ing rate

The step size $\eta$ that scales each opti­mizer update. Its peak, warmup and decay make up the sched­ule. Sec­tion 5.2

Lemon node

A node that repeat­edly fails jobs with­out fail­ing any single health check, found from his­tory. Sec­tion 12.4

Length and PMI nor­mal­iza­tion

Cor­rec­tions to cloze scores for answer length (per token or per char­ac­ter) and for the answer's prob­a­bil­ity with­out the ques­tion. Sec­tion 10.2

Length bias

A reward model's pref­er­ence for longer out­puts regard­less of qual­ity. Sec­tion 8.6

Lethal tri­fecta

An agent that com­bines pri­vate data, expo­sure to untrusted con­tent and the abil­ity to com­mu­ni­cate exter­nally. Sec­tion 14.1

Like­li­hood dis­place­ment

The chosen response becom­ing less likely during pref­er­ence train­ing while the margin over the rejected response grows. Sec­tion 8.7

Lin­eage

Recorded links from a model ver­sion to the code, con­fig­u­ra­tion, data state and run seg­ments that pro­duced it. Sec­tion 12.9

Linear atten­tion

Atten­tion with a fea­ture-map kernel, com­putable as a recur­rence with a fixed-size state. Sec­tion 4.8

Linux Secu­rity Module (LSM)

Kernel hooks for manda­tory access con­trol, for exam­ple AppArmor, SELinux and Land­lock. Sec­tion 14.1

Little's law

In a stable system, the mean number of items equals arrival rate times mean time in the system. Sec­tion 14.7

Live bench­mark

A bench­mark that keeps adding items dated after models' train­ing cut­offs. Sec­tion 10.5

LLM-as-a-judge

A model that grades or com­pares out­puts, with mea­sur­able posi­tion, ver­bosity and self-pref­er­ence biases. Sec­tion 10.6

Logit soft-cap­ping

Pass­ing logits through $c\tanh (z/c)$, a smooth bound that also shrinks the gra­di­ent of sat­u­rated logits. Gemma 2 caps atten­tion logits at 50 and output logits at 30. Sec­tion 4.3

Logits

The unnor­mal­ized scores, one per vocab­u­lary entry, that the output head pro­duces. A soft­max turns them into next-token prob­a­bil­i­ties. Sec­tion 4.1

LoRA

Low-rank adap­ta­tion: a frozen weight plus a trained low-rank update $(\alpha /r)BA$, where the rank $r$ is the inner dimen­sion of $B$ and $A$. Sec­tion 8.3

Loss aggre­ga­tion

How token losses are com­bined (global token mean, per-exam­ple mean or sum). The choice sets how much each exam­ple counts. Sec­tion 8.2

Loss mask­ing

Zero­ing the loss on tokens the model should not learn to pro­duce, such as system and user turns. Sec­tion 8.2

Loss scal­ing

Mul­ti­ply­ing the loss by a con­stant so FP16 gra­di­ents stay above the small­est rep­re­sentable value, then divid­ing it out before the update. Dynamic schemes halve the scale on over­flow. Sec­tion 5.6

Loss spike

A sudden jump in train­ing loss. Some runs recover on their own, others need a rewind to an ear­lier check­point and a data skip. Sec­tion 5.5

LR sen­si­tiv­ity

The aver­age gap to the best final loss over a wide learn­ing-rate sweep (Worts­man et al.). A flat curve means a robust recipe. Sec­tion 5.5

LSH band­ing

Group­ing MinHash values into bands so a pair becomes a can­di­date dupli­cate when every value in some band matches. Sec­tion 3.3

Major­ity vote (maj@k, cons@k)

Accu­racy of the most common final answer among k sam­ples. Sec­tion 10.3

Masked dif­fu­sion lan­guage model

A bidi­rec­tional model trained to fill ran­domly masked text and sam­pled by iter­a­tive unmask­ing. Sec­tion 4.8

Master weights

An FP32 copy of the weights that receives the opti­mizer's updates while the for­ward and back­ward passes run in lower pre­ci­sion. Sec­tion 5.6

Max­i­mum atten­tion logit

The largest pre-soft­max atten­tion score per head in a batch. Its growth sig­nals atten­tion col­lapse. Sec­tion 12.5

MaxVio

The rel­a­tive over­load of a mix­ture-of-experts layer's busiest expert: max­i­mum load minus mean load, divided by mean load. Sec­tion 4.7

McNemar's test

A paired test for 0/1 scores that uses only the ques­tions where the two models dis­agree. Sec­tion 10.4

Mean time to fail­ure (MTTF)

Expected time between fail­ures of a job. With inde­pen­dent node fail­ures at $r_f$ per node-day, $M=1/(n_{\text{nodes}}{r}_f)$: about 3.1 hours for Llama 3 405B on 16,384 GPUs. Sec­tion 12.1

Mem­ber­ship infer­ence (Min-K% Prob)

Guess­ing whether a text was in train­ing from its token prob­a­bil­i­ties, for exam­ple the mean log-prob­a­bil­ity of its least likely 20% of tokens. Sec­tion 10.5

Microan­neal

A short anneal (about 5⁠–⁠10B tokens) on a can­di­date dataset mixed with web data, com­pared against a web-only anneal, to mea­sure the dataset's value. Sec­tion 7.3

Microscal­ing (MX)

OCP for­mats in which each block of 32 ele­ments shares one 8-bit power-of-two (E8M0) scale: MXFP8, MXFP6 and MXFP4. Sec­tion 2.3

microVM

A VM with a min­i­mal device model and kernel that boots in about a hun­dred mil­lisec­onds. Sec­tion 14.2

Mid­train­ing

A train­ing stage between bulk pre­train­ing and post-train­ing with its own curated data mix, usu­ally during or right before the final learn­ing-rate decay. Also used for capa­bil­ity stages on a fin­ished base and for long-con­text stages. Sec­tion 7.2

MinHash

The min­i­mum of a random hash over a set. Two sets' MinHashes match with prob­a­bil­ity equal to their Jac­card sim­i­lar­ity. Sec­tion 3.3

Min­i­mum detectable dif­fer­ence (MDD)

The small­est gap a test calls real at a chosen sig­nif­i­cance. For seed noise $\sigma$ and $n$ seeds per arm, a two-sided 95% test needs $1.96\sigma (2/n{)}^{1/2}$. With a power require­ment added it is also called the min­i­mum detectable effect (Sec­tion 10.4). Sec­tion 15.1

Mixed pre­ci­sion

Com­put­ing in BF16 or FP16 while the opti­mizer keeps an FP32 master copy of the weights. Sec­tion 5.6

Mix­ture of experts (MoE)

A layer of many MLP experts with a router that sends each token to a few of them. Sec­tion 4.7

ML Pro­duc­tiv­ity Good­put

Google's prod­uct of sched­ul­ing good­put, run­time good­put and pro­gram good­put (MFU). Sec­tion 12.8

MLP width and the 2/3 cor­rec­tion

$d_{\text{ff}}=8d/3$ gives a three-matrix SwiGLU MLP the param­e­ters of a two-matrix MLP of width $4d$. Llama 3 uses $3.5d$. Sec­tion 4.2

MMLU, GSM8K and HumanEval

Widely reported bench­marks: mul­ti­ple-choice ques­tions on 57 sub­jects, grade-school math word prob­lems, and Python func­tions graded by unit tests. Sec­tion 10.1

Model band­width uti­liza­tion (MBU)

Achieved bytes per second (weights plus KV per output token over TPOT) divided by peak memory band­width. Sec­tion 13.2

Model-based qual­ity filter

A clas­si­fier or lan­guage-model score used to decide which doc­u­ments to keep. Sec­tion 3.2

Model col­lapse

Degra­da­tion from train­ing on model output, unbounded when syn­thetic data replaces real data and bounded when it accu­mu­lates along­side it. Sec­tion 3.6

Model Con­text Pro­to­col (MCP)

A JSON-RPC 2.0 pro­to­col through which appli­ca­tions expose tools, resources and prompts to models. It is an inter­face, not an iso­la­tion layer. Sec­tion 14.4

Model FLOPs uti­liza­tion (MFU)

Achieved model FLOP/s, exclud­ing recom­pu­ta­tion, divided by the hard­ware's dense peak. State the peak, the atten­tion con­ven­tion and the time base with every value. Sec­tion 1.3

Model ladder

A set of small train­ing runs over sizes and token bud­gets, used to fit a scal­ing law and extrap­o­late to a larger target. Sec­tion 6.4

Model soup

The aver­age of the weights of sev­eral models that share an ini­tial­iza­tion, such as anneals on dif­fer­ent data orders. Sec­tion 7.3

Model states

Weights, gra­di­ents and opti­mizer states. Sec­tion 1.4

Moment hori­zon

An expo­nen­tial moving aver­age with coef­fi­cient $\beta$ aver­ages over about $1/(1-\beta )$ steps: 20 for ${\beta }_2=0.95$, 1,000 for 0.999. Sec­tion 5.1

Mon­i­tora­bil­ity tax

The capa­bil­ity or effi­ciency given up by keep­ing opti­miza­tion pres­sure off the chain of thought. Sec­tion 9.9

MUFU (multi-func­tion unit)

The unit for expo­nen­tials, log­a­rithms and recip­ro­cals: 16 oper­a­tions per clock per SM on H100 and B200, 32 on B300. Sec­tion 2.4

Multi-head atten­tion (MHA)

One key-value head per query head, the orig­i­nal Trans­former atten­tion. Sec­tion 4.5

Multi-head latent atten­tion (MLA)

Caching one low-rank latent per token and layer, plus a small rotary key, and absorb­ing the up-pro­jec­tions into the query and output pro­jec­tions. DeepSeek-V3 caches 70,272 bytes per token. Sec­tion 4.5

Multi-query atten­tion (MQA)

All query heads share a single key-value head, the extreme case of GQA. Sec­tion 4.5

Multi-token pre­dic­tion (MTP)

An aux­il­iary objec­tive that pre­dicts tokens beyond the next one. DeepSeek-V3's extra module can also serve as a spec­u­la­tive-decod­ing draft. Sec­tion 4.8

Mul­ti­ple-choice for­mu­la­tion (MCF)

Show­ing let­tered options and scor­ing only the letter the model pre­dicts. Sec­tion 10.2

Mul­ti­ply-add (MAC, FMA)

A mul­ti­ply fol­lowed by an add, counted as 2 FLOPs. Sec­tion 1.2

Muon

SGD with momen­tum whose update matrix is orthog­o­nal­ized by a few Newton-Schulz steps. Used for hidden weight matri­ces, with AdamW for the rest. Sec­tion 5.7

μP (muP, Max­i­mal Update Param­e­ter­i­za­tion)

Per-layer ini­tial­iza­tion, learn­ing-rate and output mul­ti­pli­ers chosen so every layer's fea­tures change by a width-inde­pen­dent amount per step. Sec­tion 5.4

μTrans­fer (muTransfer)

Tuning hyper­pa­ram­e­ters on a small μP proxy model and reusing them unchanged on the large target. Sec­tion 5.4

n-gram over­lap audit

Flag­ging test items that share long token sequences with the train­ing data and com­par­ing flagged and clean scores. Sec­tion 10.5

Names­pace

Kernel fea­ture that gives a process its own view of a global resource such as mounts, process IDs, the net­work or user IDs. Sec­tion 14.1

Nat­u­ral and strat­i­fied mix­tures

Weights pro­por­tional to domain size, and equal weight per domain. Sec­tion 3.5

NCCL

NVIDIA's col­lec­tive com­mu­ni­ca­tion library, with ring, tree, NVLS and PAT algo­rithms and Simple, LL and LL128 pro­to­cols. Sec­tion 11.1

Needle in a haystack (NIAH)

A retrieval test that hides one sen­tence in filler text and sweeps con­text length against inser­tion depth. Sec­tion 7.5

Newton-Schulz iter­a­tion

A matrix-mul­ti­ply-only poly­no­mial iter­a­tion that pushes all sin­gu­lar values of a matrix toward 1 with­out chang­ing its sin­gu­lar vec­tors. Sec­tion 5.7

NF4

4-bit NormalFloat: 16 code values at quan­tiles of a normal dis­tri­bu­tion, scaled per block of 64 weights. Sec­tion 8.3

Node-lim­ited rout­ing

A cap on how many nodes one token's experts may span (4 in DeepSeek-V3), so each node receives the token once over the net­work. Sec­tion 11.6

Noise floor

The excess loss SGD keeps at a con­stant learn­ing rate, pro­por­tional to the learn­ing rate. Sec­tion 7.1

Non-embed­ding param­e­ters

Param­e­ters exclud­ing the token and posi­tion embed­ding tables. Kaplan et al.'s $N$ counts only these. Sec­tion 1.1

NoPE

No explicit posi­tional encod­ing. The causal mask still car­ries order. Sec­tion 4.6

NTK-aware scal­ing

Rais­ing the RoPE base to ${\theta }_{\text{base}}{k}_{\text{ext}}^{d_h/(d_h-2)}$ so the fastest pair is unchanged and the slow­est is slowed by exactly $k_{\text{ext}}$. Sec­tion 7.4

NVFP4

NVIDIA's 4-bit float with 16-ele­ment blocks, E4M3 block scales and an FP32 per-tensor scale: 4.5 bits per value. Sec­tion 2.3

NVLink domain (scale-up domain)

The accel­er­a­tors fully con­nected by the fast fabric: 8 on HGX, 72 on GB200 NVL72, a whole pod on TPU ICI. Sec­tion 2.5

NVLink, NVSwitch

NVIDIA's GPU-to-GPU link, and the switch chip that makes a node all-to-all. Sec­tion 2.5

NVLS (NVLink SHARP)

Reduc­tions com­puted inside NVSwitch instead of on the GPUs. Sec­tion 2.5

Obfus­cated reward hack­ing

Reward hack­ing that con­tin­ues while the chain of thought stops reveal­ing it. Sec­tion 9.9

Obser­va­tional scal­ing law

A law fit across many exist­ing models through a low-dimen­sional capa­bil­ity space instead of a ladder. Sec­tion 6.7

Occu­pancy

Res­i­dent warps per SM divided by the max­i­mum (64 on H100). Modern ker­nels often trade it for large reg­is­ter tiles. Sec­tion 2.1

OCI run­time spec

The stan­dard for a con­tainer's con­fig­u­ra­tion and life­cy­cle, imple­mented by runc, gVisor's runsc and Kata. Sec­tion 14.1

On-policy and off-policy

Responses gen­er­ated by the policy being trained versus by some other model. Sec­tion 8.8

On-policy dis­til­la­tion

The stu­dent gen­er­ates and the teacher's token prob­a­bil­i­ties super­vise each of its tokens. Sec­tion 9.4

Online and off­line

Fresh labels during train­ing versus a dataset fixed before train­ing. Sec­tion 8.8

Oracle solu­tion

A ref­er­ence solu­tion that must pass a task's tests, used to check the task is solv­able. Sec­tion 14.4

Out­come valid­ity and task valid­ity

Whether the grader truly indi­cates suc­cess, and whether a task is solv­able exactly when the target capa­bil­ity is present. Sec­tion 10.7

Out­lier fea­tures

A few hidden dimen­sions with very large acti­va­tions that make per-tensor 8-bit or 4-bit scales inac­cu­rate. Sec­tion 13.8

Output (reordered) norm

Nor­mal­iz­ing a sub­layer's output before adding it to the stream, as in OLMo 2. Sec­tion 4.3

Over-evic­tion

Evict­ing a whole sus­pect par­al­lel group instead of pin­point­ing one machine while thou­sands of GPUs wait. Sec­tion 12.4

Over­com­mit

Allo­cat­ing more vir­tual CPU or memory than phys­i­cally exists, bet­ting on low aver­age use. Sec­tion 14.8

Over­long shap­ing

Mask­ing trun­cated answers (over­long fil­ter­ing) or penal­iz­ing length lin­early just before the cap (soft over­long pun­ish­ment). Sec­tion 9.5

Over­sub­scrip­tion

The ratio of down­link to uplink capac­ity at a switch tier (Llama 3: 1:7 above each 3,072-GPU pod). Sec­tion 2.6

Over­train­ing

Train­ing past the com­pute-opti­mal tokens per param­e­ter. It is not over­fit­ting, since data is not repeated. Sec­tion 6.5

Packet spray­ing

Choos­ing a path per packet and reorder­ing at the receiver, which removes flow col­li­sions. Sec­tion 2.6

Pack­ing

Con­cate­nat­ing sev­eral exam­ples into one fixed-length row, with masks and posi­tion ids that keep them inde­pen­dent. Sec­tion 8.2

PagedAttention, KV block, block table

KV memory split into fixed-size blocks allo­cated on demand, with a per-request table map­ping log­i­cal to phys­i­cal blocks. Sec­tion 13.4

Paired dif­fer­ence

The per-ques­tion score dif­fer­ence between two models on the same ques­tions. It removes shared ques­tion dif­fi­culty. Sec­tion 10.4

Par­al­lel block

Com­put­ing atten­tion and the MLP from the same nor­mal­ized input and adding both to the stream. Sec­tion 4.4

Par­tial roll­out

Cap­ping the tokens gen­er­ated per RL iter­a­tion and resum­ing unfin­ished answers in the next one, so a step does not wait for the long­est answer. Sec­tion 9.8

pass@k

The prob­a­bil­ity that at least one of $k$ sam­pled answers passes the checker, aver­aged over prob­lems. With $K$ sam­ples of which $c$ are cor­rect, $1-\left(\genfrac{}{}{0ex}{}{K-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{K}{k}\right)$ esti­mates it with­out bias. Sec­tion 10.3

pass^k

The prob­a­bil­ity that all $k$ inde­pen­dent trials of a task suc­ceed, aver­aged over tasks and esti­mated by $\left(\genfrac{}{}{0ex}{}{c}{k}\right)/\left(\genfrac{}{}{0ex}{}{K}{k}\right)$. It mea­sures reli­a­bil­ity where pass@k mea­sures cov­er­age. Sec­tion 10.3

Patch

In byte-level models, a group of bytes pro­cessed as one step by the large global net­work. Sec­tion 3.8

Per­plex­ity

$e$ raised to the mean loss in nats per token: the effec­tive number of equally likely next tokens. It depends on the tok­enizer. Sec­tion 4.1

PF-day

One petaFLOP/s sus­tained for a day, also writ­ten PF/s-day: $8.64\times {10}^{19}$ FLOPs. Sec­tion 1.2

PFC, DCQCN

Pri­or­ity flow con­trol sends per-class pause frames that make Eth­er­net loss­less. DCQCN is ECN-driven sender rate con­trol for RoCE. Sec­tion 2.6

Pipe­line par­al­lel­ism (PP)

Split the layer stack into stages on dif­fer­ent GPUs and stream micro-batches through them. Sec­tion 11.4

Posi­tion Inter­po­la­tion (PI)

Divid­ing every rotary fre­quency by the exten­sion factor $k_{\text{ext}}$, which squeezes new posi­tions into the trained range. Sec­tion 7.4

Post-train­ing

Any train­ing after pre­train­ing: SFT, pref­er­ence tuning and RL. Sec­tion 8.1

Power law

$y=a{x}^{-k}$, a straight line of slope $-k$ on log-log axes. Sec­tion 6.1

PPO clipped sur­ro­gate

An objec­tive that stops a token's gra­di­ent once its prob­a­bil­ity ratio leaves 1 plus or minus epsilon in the help­ful direc­tion. Sec­tion 8.5

Pre-decay check­point

A check­point saved before the final learn­ing-rate decay, the pre­ferred start for con­tin­ued pre­train­ing. Sec­tion 7.6

Pre-norm / post-norm

Nor­mal­iz­ing a sub­layer's input (pre-norm) or the stream after the resid­ual add (post-norm). Sec­tion 4.3

Pre-tok­enizer

The regex that splits text into chunks before BPE, so merges never cross chunk bound­aries. Sec­tion 3.7

Pre­con­di­tioned sharp­ness

The top eigen­value of the Hes­sian after Adam's per-coor­di­nate rescal­ing. Train­ing is unsta­ble once the step size exceeds about 2 divided by it. Sec­tion 5.8

Pre­dic­tion-pow­ered infer­ence (PPI)

An esti­ma­tor that cor­rects a judge's labels with a small human-labelled sample and keeps valid con­fi­dence inter­vals. Sec­tion 10.6

Pre­emp­tion (swap or recom­pute)

Evict­ing a run­ning request's KV under memory pres­sure and later restor­ing it from CPU memory or by recom­put­ing it. Sec­tion 13.4

Pre­fill

The par­al­lel for­ward pass over the prompt that fills the KV cache and pro­duces the first output token. Sec­tion 13.1

Pre­fill-decode dis­ag­gre­ga­tion

Run­ning pre­fill and decode on sep­a­rate GPU pools and moving each request's KV cache between them. Sec­tion 13.5

Prefix caching

Reusing the KV cache of a shared prompt prefix across requests. The hit rate is cached prompt tokens over prompt tokens. Sec­tion 13.4

Pre­flight

Check­ing before train­ing that each task's ref­er­ence passes and an empty patch fails under the job's exact con­di­tions. Sec­tion 14.8

Pre­train­ing

The first and most expen­sive stage: next-token pre­dic­tion on tril­lions of tokens of mostly web text. It sets most of what the model knows. Sec­tion 0.1

Pro­mo­tion gate

A rule over tracked eval­u­a­tions that decides whether a check­point moves to the next stage. Sec­tion 12.9

Proxy and gold reward

The learned reward being opti­mized, versus the true objec­tive it stands in for. Sec­tion 8.6

Proxy model

A smaller or shorter run used to make a deci­sion, such as a data mix, for a larger one. Sec­tion 3.9

PUE

Power usage effec­tive­ness: total facil­ity power divided by the power drawn by the IT hard­ware. 1.0 is ideal. Sec­tion 2.7

QK-Clip

Kimi K2's fix for explod­ing atten­tion logits under Muon: after an opti­mizer step, shrink the query and key weights of any head whose max­i­mum logit exceeded a thresh­old. It is part of MuonClip. Sec­tion 5.7

QK-norm

Nor­mal­iz­ing queries and keys per head before their dot prod­uct, which bounds every atten­tion logit near $d_h^{1/2}$ times two gains. Sec­tion 4.3

QLoRA

LoRA on a frozen 4-bit NormalFloat (NF4) base with double quan­ti­za­tion and paged opti­mizer states. A 70B fine-tune fits one 80 GB GPU. Sec­tion 8.3

Queue pair (QP)

An RDMA con­nec­tion end­point. Extra queue pairs per GPU pair give ECMP more flows to spread. Sec­tion 2.6

RadixAttention

SGLang's prefix cache kept as a radix tree with least-recently-used leaf evic­tion and cache-aware sched­ul­ing. Sec­tion 13.4

Rail

All GPUs with the same local index across nodes, wired to the same leaf switches. Sec­tion 2.6

Rashomon effect

Many near-equally good fits that extrap­o­late very dif­fer­ently. Sec­tion 6.4

RDMA, GPUDirect RDMA

NIC-driven memory-to-memory trans­fer that bypasses the oper­at­ing system, and its form that reads and writes GPU memory directly. Sec­tion 2.6

Re-warm­ing

Rais­ing the learn­ing rate again at the start of con­tin­ued pre­train­ing, which restores plas­tic­ity and costs some old-data loss. Sec­tion 7.6

Rea­son­ing bound­ary

The set of prob­lems a model solves with nonzero prob­a­bil­ity, mea­sured by pass@k at large k. Sec­tion 9.7

Rea­son­ing model

A model trained, usu­ally with RL on ver­i­fi­able rewards, to write a long chain of thought before its answer, as DeepSeek-R1 does. Sec­tion 9.4

Recov­er­able and non-recov­er­able spike

A loss spike that returns to trend, versus one after which the model diverges or plateaus higher. Sec­tion 15.2

Reduce-scat­ter

Sum arrays across ranks and leave each rank with only its 1/p slice of the sum. Sec­tion 11.1

Reduc­ible loss

Loss minus the floor, read as the KL diver­gence from model to data. It is the part that falls with scale. Sec­tion 6.1

Ref­er­ence policy

The frozen model, usu­ally the SFT model, that KL and log-ratios are mea­sured against. Sec­tion 8.5

Ref­er­ence (pos­i­tive) set

The text a qual­ity clas­si­fier is trained to rec­og­nize as good. Sec­tion 3.2

Ref­er­ence reset

Replac­ing the KL ref­er­ence policy with the cur­rent policy during train­ing. Sec­tion 9.6

Rehy­dra­tion

Upsam­pling doc­u­ments by their stored dupli­cate-clus­ter size after global dedu­pli­ca­tion, as in FineWeb2. Sec­tion 3.3

Rein­force­ment learn­ing from human feed­back (RLHF)

Fine-tuning with rein­force­ment learn­ing against a reward model trained on human pref­er­ence com­par­isons, usu­ally with a KL penalty to a ref­er­ence policy. Sec­tion 8.5

Rejec­tion sam­pling

Sam­pling sev­eral responses, keep­ing the best by a reward model or ver­i­fier, and fine-tuning on it. Sec­tion 8.5

Replay

Mixing a share of old (or sim­i­lar) data into every batch during con­tin­ued pre­train­ing to limit for­get­ting. Com­pute-equiv­a­lent replay keeps the total token count fixed. Sec­tion 7.6

Request vs limit

Kuber­netes's guar­an­teed allo­ca­tion used for sched­ul­ing versus the enforced ceil­ing (CPU throt­tling, memory OOM kill). Sec­tion 14.8

Reshard­ing

Load­ing a check­point into a dif­fer­ent par­al­lel layout than the one that saved it. Sec­tion 12.3

Resid­ual states

Acti­va­tions, tem­po­rary buf­fers and frag­mented memory. Sec­tion 1.4

Resid­ual stream

The run­ning sum that every block reads from and adds its output to. Sec­tion 4.2

Restart cost

$R$: time per fail­ure to detect, diag­nose, resched­ule, ini­tial­ize, load the check­point and warm up. Sec­tion 12.3

Reward hack­ing

Rais­ing the mea­sured reward with­out doing the intended task, a form of spec­i­fi­ca­tion gaming. Reward tam­per­ing changes the reward mech­a­nism itself. Sec­tion 14.6

Reward model

A net­work that maps a prompt and response to a scalar, trained so that score dif­fer­ences pre­dict human pref­er­ences. Sec­tion 8.4

Reward over-opti­miza­tion

True qual­ity fall­ing while the proxy reward keeps rising (Good­hart's law). Sec­tion 8.6

Rewind-and-skip

Restart­ing from a check­point shortly before a loss spike and skip­ping the data batches seen around it. Sec­tion 5.5

Ridge point

Peak FLOP/s divided by band­width: about 295 FLOP per byte for an H100 in dense BF16. For decode it is also the batch at which weight mat­muls turn com­pute-bound. Sec­tion 1.7

Ring all-reduce

Reduce-scat­ter then all-gather around a log­i­cal ring: $2(p-1)$ steps, with $2(p-1)/p$ of the data sent per rank. Sec­tion 11.1

Ring Atten­tion

Con­text par­al­lel­ism that passes key and value blocks around a ring of GPUs while each com­putes block­wise atten­tion, hiding the trans­fer when blocks are large enough. Sec­tion 11.5

River-valley land­scape

A pic­ture of the loss as a steep valley with a slowly descend­ing floor: a high learn­ing rate moves along the floor while bounc­ing on the walls, and decay removes the bounce. Sec­tion 5.2

RL readi­ness

How well a base model improves under rein­force­ment learn­ing, which mid­train­ing data can shape. Sec­tion 7.2

RLAIF

Rein­force­ment learn­ing from AI feed­back: pref­er­ence labels pro­duced by a model instead of people. Sec­tion 8.4

RLOO

REINFORCE leave-one-out: advan­tage = reward minus the mean reward of the other G - 1 answers. Sec­tion 9.3

RLVR

Rein­force­ment learn­ing with ver­i­fi­able rewards: RL where a pro­gram, not a learned model, com­putes the reward. Sec­tion 9.1

RMSNorm

Nor­mal­iza­tion by the root mean square of the fea­tures with a learned gain and no mean sub­trac­tion. Sec­tion 4.3

Roll­out

A response sam­pled from the cur­rent policy during RL and then scored by a reward. Gen­er­at­ing roll­outs is decode-bound infer­ence. Sec­tion 9.2

Roll­out rout­ing replay (R3)

Reusing the infer­ence engine's mix­ture-of-experts rout­ing when the trainer recom­putes prob­a­bil­i­ties. Sec­tion 9.6

Roofline model

Attain­able FLOP/s equals the min­i­mum of peak FLOP/s and band­width times arith­metic inten­sity. Sec­tion 1.7

RoPE (rotary posi­tion embed­ding)

Rotat­ing each query and key pair by an angle pro­por­tional to posi­tion, so atten­tion scores depend only on the rel­a­tive offset. The base ${\theta }_{\text{base}}$ sets the rota­tion fre­quen­cies: Llama 3 uses 500,000. Sec­tion 4.6

Round-to-near­est (RTN), post-train­ing quan­ti­za­tion (PTQ)

The naive quan­tizer, and quan­tiz­ing a trained model with a small cal­i­bra­tion set and no retrain­ing. Sec­tion 13.8

Router and gate value

The small linear layer that scores experts, and the weight applied to a chosen expert's output. Sec­tion 4.7

Router z-loss

The squared log-sum-exp of router logits, added to keep them small (ST-MoE). Sec­tion 4.7

Run­ning exam­ple

One of the pub­lished models (Llama 3 8B, 70B and 405B, DeepSeek-V3 and R1, GPT-2 small, OLMo 2) whose num­bers this guide reuses in every chap­ter. Sec­tion 0.3

RuntimeClass

Kuber­netes object that selects the con­tainer run­time (for exam­ple runc, gVisor or Kata) per pod. Sec­tion 14.2

Sand­box

An iso­lated envi­ron­ment, such as a con­tainer, a gVisor sand­box or a microVM, where model-writ­ten code or agent actions run during RL, eval­u­a­tion or deploy­ment. Sec­tion 14.2

Sand­wich norm (Peri-LN)

Nor­mal­iz­ing both the input and the output of each sub­layer, as in Gemma 2 and 3. Sec­tion 4.3

Sat­u­ra­tion (bench­mark)

Loss of reli­able dis­crim­i­na­tion among the top models. Akhtar et al.'s index is $e^{-R^2}$, with $R$ the top-five spread in stan­dard errors. Sec­tion 10.1

Scaf­fold

The pro­gram around the model in an agent: prompts, tools, his­tory man­age­ment and retries. An agent's score belongs to the model and its scaf­fold together. Sec­tion 10.7

Scal­ing factor

A higher-pre­ci­sion mul­ti­plier that maps a tensor, row or block into a narrow format's range. Sec­tion 2.3

Scal­ing law

A fitted rela­tion, usu­ally a power law, between loss or accu­racy and param­e­ters, data or com­pute, used to size runs before train­ing them. Sec­tion 6.1

Sched­ule-free opti­mizer

An opti­mizer that removes the decay by eval­u­at­ing a run­ning aver­age of iter­ates and taking gra­di­ents at an inter­po­la­tion. Sec­tion 5.7

Sci­en­tific, nui­sance and fixed hyper­pa­ram­e­ters

The hyper­pa­ram­e­ter under study, those re-tuned for each of its values so the com­par­i­son is fair, and those held con­stant. Sec­tion 15.1

Score-func­tion esti­ma­tor

The iden­tity $\nabla 𝔼[R]=𝔼[R\nabla \log \pi ]$, which gives a policy gra­di­ent with­out dif­fer­en­ti­at­ing the reward. It is also called the log-deriv­a­tive trick. Sec­tion 9.2

sec­comp-bpf

A system-call filter writ­ten as a small BPF pro­gram, inher­ited by child pro­cesses. Sec­tion 14.1

Seed vari­ance

Run-to-run spread of a metric at fixed hyper­pa­ram­e­ters, from ini­tial­iza­tion, data order and non­de­ter­min­ism. Sec­tion 10.9

Selec­tive batch­ing

Run­ning token-wise oper­a­tions on the flat­tened tokens of all requests and atten­tion per request. Sec­tion 13.4

Selec­tive recom­pu­ta­tion

Recom­put­ing only the memory-heavy, cheap-to-recom­pute parts of a layer, such as the atten­tion matrix. Sec­tion 1.5

Sep­a­ra­bil­ity

Share of model pairs whose bench­mark con­fi­dence inter­vals do not over­lap. Sec­tion 10.6

Sequence par­al­lel­ism (SP)

The com­pan­ion of TP that splits layer norms and dropout along the sequence, turn­ing each all-reduce into a reduce-scat­ter and an all-gather of the same total size. Sec­tion 11.3

Ser­vice-level objec­tive (SLO)

A latency target such as TTFT under 2 s and TPOT under 50 ms at the 99th per­centile. Sec­tion 13.3

Sham­poo

A pre­con­di­tioner built from left and right Kro­necker fac­tors of accu­mu­lated gra­di­ent outer prod­ucts. Sec­tion 5.7

Shared expert

An expert that every token always uses, beside its routed experts. Sec­tion 4.7

Shared memory (SMEM)

Per-SM, soft­ware-man­aged on-chip memory: up to 228 KB per SM on an H100, read at 128 bytes per clock. Sec­tion 2.2

Shin­gle

A word n-gram used to rep­re­sent a doc­u­ment as a set (word 5-grams in FineWeb). Sec­tion 3.3

Sig­moidal RL scal­ing

Expected reward against RL com­pute fitted with a bounded curve with a ceil­ing, a steep­ness and a mid­point. Sec­tion 6.8

Signal-to-noise ratio (eval­u­a­tion)

The spread of a metric across can­di­date models trained at the same com­pute, divided by one model's check­point-to-check­point noise. High-SNR met­rics give better small-scale deci­sions. Sec­tion 10.9

Silent data cor­rup­tion (SDC)

Hard­ware return­ing a wrong result with­out any error signal. A CPU core that does this is called a mer­cu­rial core. Sec­tion 12.2

Slid­ing-window atten­tion

Each token attends only to the pre­vi­ous w tokens. A rolling buffer stores just w cache slots. Sec­tion 4.5

Slow diver­gence

An insta­bil­ity that wors­ens final loss at high learn­ing rates with­out a vis­i­ble spike. Sec­tion 15.2

Snap­shot and restore

Saving a run­ning sand­box's memory and device state and resum­ing it later, pos­si­bly as many clones. Sec­tion 14.3

SOAP

Adam run in the eigen­ba­sis of Sham­poo's pre­con­di­tioner, refreshed every few steps. Sec­tion 5.7

Sparse upcy­cling

Ini­tial­iz­ing an MoE from a dense check­point by copy­ing its MLP into every expert. Sec­tion 4.7

Spe­cial tokens

Reserved token ids for con­trol strings such as end-of-text or chat head­ers. Sec­tion 3.7

Spec­u­la­tive decod­ing

A cheap draft pro­poses sev­eral tokens, the target model scores them in one pass, and an accept-or-resam­ple rule keeps the output dis­tri­bu­tion exactly the target's. Sec­tion 13.7

Spike score

OLMo 2's sta­bil­ity metric: the per­cent­age of values at least 7 stan­dard devi­a­tions from the rolling mean of the pre­vi­ous 1,000. Sec­tion 5.5

Split-KV decod­ing (Flash-Decod­ing)

Par­al­leliz­ing decode atten­tion over the cached sequence and merg­ing par­tial soft­maxes exactly with their log-sum-exp. Sec­tion 13.8

Spu­ri­ous reward

A reward unre­lated to cor­rect­ness, such as a random or format-only reward. Sec­tion 9.7

Square-root scal­ing rule

When the batch grows by $\kappa$, mul­ti­ply Adam's learn­ing rate by ${\kappa }^{1/2}$. SGD's linear rule mul­ti­plies it by $\kappa$. Sec­tion 5.3

Stack aggre­ga­tion

Group­ing the stacks of all ranks during a hang so the odd one out iden­ti­fies the cul­prit. Sec­tion 12.6

Stale second moment

Adam's second-moment esti­mate lag­ging behind a sudden rise in squared gra­di­ents, which pro­duces an over­sized update. Sec­tion 5.5

Stal­e­ness

The number of policy updates between the policy that sam­pled a token and the policy being trained. Sec­tion 9.8

Stall-free batch­ing

Sarathi-Serve's policy: fill each iter­a­tion with all ongo­ing decodes first, then pre­fill chunks up to the token budget. Sec­tion 13.5

Stan­dard error (SE)

Stan­dard devi­a­tion of an esti­mate over hypo­thet­i­cal resam­ples of the ques­tions. Sec­tion 10.4

Stan­dard param­e­ter­i­za­tion (SP)

Fan-in ini­tial­iza­tion and one global learn­ing rate, the default in deep-learn­ing frame­works. Sec­tion 5.4

State space model (SSM)

A sequence layer from a dis­cretized linear dynam­i­cal system. Mamba makes its param­e­ters input-depen­dent. Sec­tion 4.8

Sta­tis­ti­cal power

The prob­a­bil­ity that a test detects a true effect of a given size. Sec­tion 10.4

Sto­chas­tic round­ing

Round­ing up or down at random with prob­a­bil­i­ties that make the rounded value equal the true value on aver­age. Sec­tion 5.6

Strag­gler (fail-slow)

A rank that keeps work­ing but slowly, so every syn­chro­nous step waits for it. Sec­tion 12.2

Stream­ing mul­ti­pro­ces­sor (SM)

The repeated pro­ces­sor of an NVIDIA GPU, with its own tensor cores, reg­is­ters, shared memory and warp sched­ulers. H100 SXM has 132. Sec­tion 2.1

Struc­tured prun­ing (width, depth)

Remov­ing whole heads, MLP neu­rons, embed­ding chan­nels (width) or layers (depth), so the result is a smaller dense model. Sec­tion 7.7

Suffix array

The sorted list of all suf­fixes of a text, used to find repeated sub­strings for exact-sub­string dedu­pli­ca­tion. Sec­tion 3.3

Super­fi­cial align­ment hypoth­e­sis

The claim that align­ment mostly teaches format while knowl­edge and capa­bil­i­ties come from pre­train­ing. Sec­tion 8.1

Super­vised fine-tuning (SFT)

Next-token train­ing on curated prompt and response pairs with loss on the responses only. Also called instruc­tion tuning. Sec­tion 8.2

Swamp­ing

An update smaller than half the spac­ing of rep­re­sentable num­bers at the weight's mag­ni­tude rounds away and is lost. Sec­tion 5.6

SWE-bench

Tasks built from real GitHub issues in Python repos­i­to­ries. An agent's patch counts only if it applies and the repos­i­tory's tests pass. Sec­tion 10.7

SwiGLU

A gated MLP, $(\mathrm{S}\mathrm{i}\mathrm{L}\mathrm{U}(x{W}_{\text{gate}})\odot x{W}_{\text{up}})W_{\text{down}}$, with three weight matri­ces. Sec­tion 4.2

Syco­phancy

Telling users what they want to hear rather than what is true. Sec­tion 8.6

Syn­thetic data

Model-writ­ten train­ing text, either gen­er­ated from prompts or rephrased from real doc­u­ments. Sec­tion 3.6

Sys­tolic array, MXU

A grid of mul­ti­ply-accu­mu­late cells that pass oper­ands to neigh­bors with­out memory access. The MXU is a TPU's sys­tolic matrix unit. Sec­tion 2.8

Task loss (bits per byte)

Neg­a­tive log-like­li­hood of the cor­rect answer divided by its length in bytes. Sec­tion 6.7

Task vector

The dif­fer­ence between fine-tuned and pre­trained weights. Adding it adds a skill, negat­ing it removes one. Sec­tion 8.3

TDP

Ther­mal design power: the sus­tained power a chip's cool­ing is designed for (700 W for an H100 SXM), not its aver­age draw. Sec­tion 2.7

Teacher cor­rec­tion

Briefly fine-tuning the teacher on the dis­til­la­tion data before dis­till­ing, when the orig­i­nal pre­train­ing data is unavail­able. Sec­tion 7.7

Tem­per­a­ture, top-k, nucleus (top-p), min-p

Logit scal­ing, and trun­ca­tion to the k like­li­est tokens, to the small­est set with cumu­la­tive prob­a­bil­ity p, or to tokens above a frac­tion of the top prob­a­bil­ity. Sec­tion 13.9

Tensor core

A matrix unit that per­forms a small matrix mul­ti­ply-accu­mu­late per instruc­tion. Tensor cores supply about 94% of an H100's arith­metic. Sec­tion 2.1

Tensor memory (TMEM)

Black­well's 256 KB per-SM memory that holds tensor-core accu­mu­la­tors. Sec­tion 2.2

Tensor par­al­lel­ism (TP)

Split each weight matrix across GPUs (column-par­al­lel then row-par­al­lel), with col­lec­tives inside every layer. Sec­tion 11.3

Ter­mi­nated vs trun­cated

An episode that ended from the task's own state versus one stopped by an out­side limit, crash or time­out. Sec­tion 14.4

Text extrac­tion

Select­ing a page's main con­tent from its HTML, with tools such as trafi­latura, resili­parse or jusText. Sec­tion 3.1

Thread block (CTA), thread-block clus­ter

Threads sched­uled on one SM that share its shared memory. A clus­ter is a set of blocks co-sched­uled on SMs of one GPC that can read each other's shared memory. Sec­tion 2.1

Tied embed­dings

One matrix used both as the input embed­ding and as the output pro­jec­tion. GPT-2 ties them and Llama 3 does not. Sec­tion 1.1

Tile and wave quan­ti­za­tion

Wasted work when a matrix dimen­sion is not a mul­ti­ple of the tile size, or the tile count not a mul­ti­ple of the number of SMs. Sec­tion 1.7

Time between tokens (TBT)

The gap between two con­sec­u­tive tokens of one stream. Its tail (p99) is what users notice as a stall. Sec­tion 13.5

Time hori­zon (50%, 80%)

The human task length at which an agent's fitted suc­cess prob­a­bil­ity is 50% (or 80%), a mea­sure of task dif­fi­culty rather than of the agent's work­ing time (METR). Sec­tion 10.7

Time per output token (TPOT), inter-token latency (ITL)

Aver­age gap between output tokens after the first. Tools differ on whether the first token is included. Sec­tion 13.3

Time to first token (TTFT)

Time from request sub­mis­sion to the first output token: queue­ing, pre­fill and net­work. Sec­tion 13.3

TMA (Tensor Memory Accel­er­a­tor)

Hopper's asyn­chro­nous engine that copies tensor tiles between HBM and shared memory. Sec­tion 2.2

Token and tok­enizer

A token is one inte­ger id from the vocab­u­lary, often a piece of a word. The tok­enizer, usu­ally byte-level BPE, maps text to tokens and back. Sec­tion 3.7

Token bound­ary bias

Dis­tor­tion when a prompt ends part­way through what train­ing would have tok­enized as a single token. Sec­tion 3.7

Token choice / expert choice

Tokens pick their top-k experts, or experts pick their top tokens from the batch. Sec­tion 4.7

Token-level loss aggre­ga­tion

Aver­ag­ing the loss over all tokens in a group instead of per response. Sec­tion 9.5

Tok­eniza­tion pre­mium

The ratio of token counts for the same con­tent in two lan­guages. Sec­tion 3.7

Tokens per param­e­ter (D/N)

Train­ing tokens divided by param­e­ters: about 20 at Chin­chilla's com­pute opti­mum and about 1,870 for Llama 3 8B. Sec­tion 6.2

Total / active param­e­ters

All stored weights, and the weights one token's for­ward pass uses. Sec­tion 4.7

Train­ing-infer­ence mis­match

Dif­fer­ent token prob­a­bil­i­ties from the infer­ence engine and the trainer for the same weights. Sec­tion 9.6

Train­ing on the test task

Using a bench­mark's format or train­ing split during train­ing. Not mal­prac­tice, but a con­found when com­par­ing models. Sec­tion 7.3

Train­ing pipe­line

The ordered stages that turn raw data into a served model: data, pre­train­ing, mid­train­ing, post-train­ing, eval­u­a­tion and serv­ing. Sec­tion 0.1

Triton, torch.com­pile

Triton is a Python-embed­ded, tile-based kernel lan­guage. torch.com­pile is PyTorch's com­piler, which fuses oper­a­tions and emits Triton. Sec­tion 2.4

Trun­cated impor­tance sam­pling

Cap­ping an impor­tance weight at a con­stant to bound its vari­ance, at the cost of bias. Sec­tion 9.6

Under-trained token

A token in the vocab­u­lary that was nearly or entirely absent from the model's train­ing data. Sec­tion 3.7

Uni­gram-nor­mal­ized loss

Per-token loss rel­a­tive to a con­text-free uni­gram model, com­pa­ra­ble across vocab­u­lary sizes. Sec­tion 3.8

Update RMS

$(𝔼[g^2/u]{)}^{1/2}$ for Adam, with $u$ its second-moment esti­mate. It is near 1 when the esti­mate is cur­rent and far from 1 when it is stale, which often pre­cedes a loss spike. Sec­tion 12.5

User names­pace

A names­pace that maps user IDs so that root inside a con­tainer is an unpriv­i­leged user on the host. Sec­tion 14.1

Value model

A net­work that pre­dicts the expected future reward at each token, used by PPO to com­pute advan­tages. Also called the critic. Sec­tion 8.5

Ver­i­fi­able instruc­tion

A con­straint in a prompt whose sat­is­fac­tion a pro­gram can check, as in IFEval. Sec­tion 10.6

Ver­i­fi­ca­tion gap

The dif­fer­ence between cov­er­age and the accu­racy a real selec­tion method (voting, reward model) achieves. Sec­tion 6.8

Vio­la­tion rate and false refusal rate

The share of responses that break a safety policy, and the share of harm­less prompts the model refuses. Sec­tion 8.9

VMGenID

A vir­tual device that tells a guest it was restored from a snap­shot so its kernel can reseed ran­dom­ness. Sec­tion 14.3

VMM

Vir­tual machine mon­i­tor: the user-space pro­gram that sets up a VM and emu­lates its devices, for exam­ple QEMU, Fire­cracker or Cloud Hyper­vi­sor. Sec­tion 14.2

Vocab­u­lary size

$V$, the number of dis­tinct tokens: 128,256 for Llama 3 and 50,257 for GPT-2. It sets the embed­ding and output-head sizes and the loss at ini­tial­iza­tion, $\ln V$. Sec­tion 3.8

WARC, WAT, WET

Common Crawl's raw HTTP responses with HTML, its com­puted meta­data (head­ers, links), and its own extracted plain text. Sec­tion 3.1

Warm pool

Started but unused sand­boxes held ready to hide cre­ation latency. Sec­tion 14.7

Warmup

A ramp of the learn­ing rate from zero to its peak over the first steps of train­ing. Sec­tion 5.2

Warmup-stable-decay (WSD)

A warmup, a long con­stant learn­ing rate, then a short cooldown. Any con­stant-phase check­point can be branched into a fin­ished model. Sec­tion 5.2

Warp spe­cial­iza­tion

A kernel design in which some warps only move data while others only com­pute. Sec­tion 2.4

Warp, warp­group

32 threads that exe­cute the same instruc­tion together (SIMT). A warp­group is 4 warps (128 threads) that issue Hopper's WGMMA together. Sec­tion 2.1

WASI

The WebAssembly System Inter­face, which gives a module only the capa­bil­i­ties its host explic­itly grants. Sec­tion 14.2

Wave­length

The tokens a rotary pair needs to com­plete one full rota­tion, ${\lambda }_j=2\pi /{\omega }_j$. Sec­tion 7.4

Weight-only and weight-acti­va­tion quan­ti­za­tion

W4A16 stores weights in low pre­ci­sion and com­putes in 16-bit. W8A8 also quan­tizes acti­va­tions, so the matmul runs in 8 bits. Sec­tion 13.8

Weight sync

Copy­ing new policy weights from the trainer into the infer­ence engine, includ­ing any reshard­ing. Sec­tion 9.8

WGMMA, tcgen05

Hopper's asyn­chro­nous warp­group matrix-mul­ti­ply instruc­tion, and Black­well's single-thread tensor-core instruc­tion family that writes to tensor memory. Sec­tion 2.4

Wilson inter­val

A bino­mial con­fi­dence inter­val that keeps close to nom­i­nal cov­er­age on small bench­marks. Sec­tion 10.4

Xid

An NVIDIA driver error code in the kernel log with a cat­a­logued mean­ing and recov­ery action. Sec­tion 12.5

YaRN

Per-pair inter­po­la­tion by turns per train­ing window (NTK-by-parts) plus an atten­tion tem­per­a­ture $(1/t{)}^{1/2}=0.1\ln k_{\text{ext}}+1$. Sec­tion 7.4

Young inter­val

The first-order opti­mal check­point inter­val ${\tau }^{\star }=(2\delta M{)}^{1/2}$ for block­ing check­point cost $\delta$ and job MTTF $M$. At the opti­mum, check­point time equals expected rework. Sec­tion 12.3

z-loss

An aux­il­iary loss ${\lambda }_z(\log Z{)}^2$ on the output soft­max nor­mal­izer $Z$ that keeps logits from drift­ing in the one direc­tion cross-entropy cannot see. Sec­tion 4.3

Zero-bubble sched­ule

A pipe­line sched­ule that splits the back­ward pass into input-gra­di­ent and weight-gra­di­ent work and uses the latter to fill the bubble. Sec­tion 11.4

ZeRO stages 1, 2 and 3

Shard­ing the opti­mizer states (stage 1), then also the gra­di­ents (stage 2), then also the param­e­ters (stage 3) across the data-par­al­lel group. Sec­tion 11.2

Zero-vari­ance group

A group whose answers all get the same reward, so every advan­tage and the gra­di­ent are zero. Sec­tion 9.3

16.3 Reading list: fifty sources to read in full

If you can read only fifty things, which ones, and why each? The 50 sources below are grouped by the parts of this guide, with the reports behind the run­ning exam­ples first. Each entry gives the title and year, one line on why it is worth your time, and the chap­ters that lean on it, so you can pick what to read next by the part you are work­ing in.

Run­ning exam­ples

• The Llama 3 Herd of Models (2024) [6]. The most com­plete public account of a fron­tier run: data, scal­ing-law sizing, 4D par­al­lel­ism on 16,384 GPUs, the fail­ure log and six rounds of post-train­ing. Used in Chap­ters 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15.
• DeepSeek-V3 Tech­ni­cal Report (2024) [2]. A mix­ture of experts with multi-head latent atten­tion, FP8 train­ing and over­lapped expert par­al­lel­ism, with GPU-hours reported by stage. Used in Chap­ters 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15.
• DeepSeek-R1: Incen­tiviz­ing Rea­son­ing Capa­bil­ity in LLMs via Rein­force­ment Learn­ing (2025) [3]. Rea­son­ing from rein­force­ment learn­ing on ver­i­fi­able rewards: R1-Zero with­out super­vised data, then the multi-stage R1 recipe and its GPU-hours. Used in Chap­ters 2, 4, 6, 7, 8, 9, 10, 13.
• 2 OLMo 2 Furi­ous (2024) [7]. A fully open run with data, code and check­points: sta­bil­ity fixes, a curated mid­train­ing mix and aver­aged anneals. Used in Chap­ters 1, 3, 4, 5, 6, 7, 8, 10, 12, 15.
• Tülu 3: Push­ing Fron­tiers in Open Lan­guage Model Post-Train­ing (2024) [550]. An open post-train­ing recipe from start to finish: SFT mixes, DPO and RL with ver­i­fi­able rewards, with abla­tions at each stage. Used in Chap­ters 8, 9, 10.

Part I. Com­pute

• Roofline: An Insight­ful Visual Per­for­mance Model for Float­ing-Point Pro­grams and Mul­ti­core Archi­tec­tures (2008) [79]. The roofline model: one plot that says whether a kernel is lim­ited by com­pute or by memory band­width. Used in Chap­ters 1, 2, 15.
• PaLM: Scal­ing Lan­guage Mod­el­ing with Path­ways (2022) [34]. Defines model FLOPs uti­liza­tion, doc­u­ments loss spikes and the restart-and-skip fix, and trains 540B param­e­ters on 6,144 TPU v4 chips. Used in Chap­ters 1, 2, 3, 4, 5, 11, 12, 15.
• FlashAttention: Fast and Memory-Effi­cient Exact Atten­tion with IO-Aware­ness (2022) [63]. IO-aware atten­tion: tiling and an online soft­max keep the atten­tion matrix out of HBM, the pat­tern behind modern fused ker­nels. Used in Chap­ters 1, 2, 7.
• How to Think About GPUs (2025) [138]. A worked tour of GPU hard­ware, from stream­ing mul­ti­pro­ces­sors and the memory hier­ar­chy to NVLink and col­lec­tives, with the arith­metic for each. Used in Chap­ters 2, 11.

Part II. Pre­train­ing

• The FineWeb Datasets: Decant­ing the Web for the Finest Text Data at Scale (2024) [5]. How a 15T-token web dataset is built and tested: extrac­tion, fil­ter­ing, dedu­pli­ca­tion and the abla­tions that jus­tify each step. Used in Chap­ter 3.
• DataComp-LM: In search of the next gen­er­a­tion of train­ing sets for lan­guage models (2024) [167]. A con­trolled bench­mark for data cura­tion, where a model-based qual­ity filter beats hand-writ­ten heuris­tics at fixed com­pute. Used in Chap­ter 3.
• Scal­ing Data-Con­strained Lan­guage Models (2023) [209]. What repeated data is worth: up to about four epochs is nearly as good as fresh tokens, with a fitted law for the decay. Used in Chap­ters 3, 6, 15.
• LLaMA: Open and Effi­cient Foun­da­tion Lan­guage Models (2023) [46]. The ref­er­ence recipe for the modern dense decoder: RMSNorm pre-nor­mal­iza­tion, SwiGLU and RoPE, trained on public data. Used in Chap­ters 1, 3, 4, 6.
• DeepSeek-V2: A Strong, Eco­nom­i­cal, and Effi­cient Mix­ture-of-Experts Lan­guage Model (2024) [282]. Intro­duces multi-head latent atten­tion and the DeepSeekMoE layer with fine-grained and shared experts. Used in Chap­ters 1, 4, 5, 7, 11, 13.
• Tensor Pro­grams V: Tuning Large Neural Net­works via Zero-Shot Hyper­pa­ram­e­ter Trans­fer (2022) [372]. Max­i­mal update parametriza­tion: tune hyper­pa­ram­e­ters on a narrow proxy and trans­fer them unchanged to a wide model. Used in Chap­ter 5.
• Small-scale prox­ies for large-scale Trans­former train­ing insta­bil­i­ties (2023) [270]. Repro­duces large-model insta­bil­i­ties in small models and shows which fixes, such as QK-norm and z-loss, pre­vent them. Used in Chap­ters 4, 5, 12, 15.

Part III. Scal­ing laws

• Scal­ing Laws for Neural Lan­guage Models (2020) [25]. The first sys­tem­atic power laws for loss against param­e­ters, data and com­pute, and the 6N FLOPs-per-token account­ing. Used in Chap­ters 1, 3, 4, 5, 6, 15.
• Train­ing Com­pute-Opti­mal Large Lan­guage Models (2022) [40]. Chin­chilla: scale param­e­ters and tokens in equal pro­por­tion, about 20 tokens per param­e­ter, from three inde­pen­dent fit­ting approaches. Used in Chap­ters 1, 5, 6, 15.
• Resolv­ing Dis­crep­an­cies in Com­pute-Opti­mal Scal­ing of Lan­guage Models (2024) [27]. Explains why Kaplan and Chin­chilla dis­agreed by trac­ing the gap to head FLOPs, warmup and hyper­pa­ram­e­ter tuning. Used in Chap­ters 1, 5, 6.
• Beyond Chin­chilla-Opti­mal: Account­ing for Infer­ence in Lan­guage Model Scal­ing Laws (2023) [70]. Adds infer­ence to the sizing prob­lem and shows why a model that will serve many tokens should be smaller and trained longer. Used in Chap­ters 1, 6, 13, 15.
• Scal­ing LLM Test-Time Com­pute Opti­mally can be More Effec­tive than Scal­ing Model Param­e­ters (2024) [71]. When extra test-time com­pute beats a larger model, and how to split it between par­al­lel sam­pling and sequen­tial revi­sion. Used in Chap­ters 1, 6, 9.

Part IV. Mid­train­ing

• Scal­ing Laws and Com­pute-Opti­mal Train­ing Beyond Fixed Train­ing Dura­tions (2024) [353]. A con­stant learn­ing rate with a short cooldown matches cosine and makes one run reus­able for many token bud­gets. Used in Chap­ters 5, 6, 7, 15.
• YaRN: Effi­cient Con­text Window Exten­sion of Large Lan­guage Models (2023) [507]. YaRN: con­text exten­sion that inter­po­lates only the slow RoPE fre­quen­cies and adds an atten­tion tem­per­a­ture. Used in Chap­ter 7.
• Simple and Scal­able Strate­gies to Con­tin­u­ally Pre-train Large Lan­guage Models (2024) [358]. Con­tin­ued pre­train­ing with­out for­get­ting: rewarm and re-decay the learn­ing rate and replay a few per­cent of the old data. Used in Chap­ters 5, 7.
• LLM Prun­ing and Dis­til­la­tion in Prac­tice: The Mini­tron Approach (2024) [544]. Mini­tron: prune a trained model and recover it with dis­til­la­tion for a frac­tion of the cost of train­ing from scratch. Used in Chap­ter 7.

Part V. Post-train­ing

• Train­ing lan­guage models to follow instruc­tions with human feed­back (2022) [8]. InstructGPT: the SFT, reward model and PPO recipe, with a 1.3B model pre­ferred to the 175B base model. Used in Chap­ter 8.
• Direct Pref­er­ence Opti­miza­tion: Your Lan­guage Model is Secretly a Reward Model (2023) [586]. DPO: pref­er­ence tuning as a clas­si­fi­ca­tion loss on log-prob­a­bil­ity ratios, with no reward model and no sam­pling loop. Used in Chap­ters 8, 15.
• Scal­ing Laws for Reward Model Overop­ti­miza­tion (2022) [575]. Mea­sures how opti­miz­ing a learned reward model first raises and then lowers the true reward, with fitted laws for both. Used in Chap­ters 6, 8, 10.
• LoRA: Low-Rank Adap­ta­tion of Large Lan­guage Models (2021) [565]. Low-rank adapters: fine-tune a large model by train­ing two small matri­ces beside each frozen weight. Used in Chap­ters 1, 8.
• DeepSeekMath: Push­ing the Limits of Math­e­mat­i­cal Rea­son­ing in Open Lan­guage Models (2024) [207]. Intro­duces GRPO, the critic-free group-base­line algo­rithm behind most rea­son­ing RL recipes. Used in Chap­ters 3, 7, 9, 15.
• DAPO: An Open-Source LLM Rein­force­ment Learn­ing System at Scale (2025) [640]. An open and ablated large-scale RL recipe: asym­met­ric clip­ping, dynamic sam­pling and a token-level loss. Used in Chap­ter 9.
• Mon­i­tor­ing Rea­son­ing Models for Mis­be­hav­ior and the Risks of Pro­mot­ing Obfus­ca­tion (2025) [733]. Chain-of-thought mon­i­tors catch reward hack­ing in coding agents, and opti­miz­ing against the mon­i­tor teaches models to hide it. Used in Chap­ters 9, 14.

Part VI. Eval­u­a­tion

• Eval­u­at­ing Large Lan­guage Models Trained on Code (2021) [699]. HumanEval and the unbi­ased pass@k esti­ma­tor, the basis of sam­pling-based code eval­u­a­tion. Used in Chap­ters 6, 9, 10, 14, 15.
• Adding Error Bars to Evals: A Sta­tis­ti­cal Approach to Lan­guage Model Eval­u­a­tions (2024) [777]. Error bars for evals: stan­dard errors, clus­tered ques­tions and paired com­par­isons for bench­mark scores. Used in Chap­ters 10, 15.
• Lessons from the Trenches on Repro­ducible Eval­u­a­tion of Lan­guage Models (2024) [775]. Prac­ti­cal lessons on repro­ducible eval­u­a­tion: prompt and format sen­si­tiv­ity, and what to report with every score. Used in Chap­ters 10, 12.
• Mea­sur­ing AI Abil­ity to Com­plete Long Soft­ware Tasks (2025) [486]. Mea­sures agents by the length of the tasks they can finish and finds that this hori­zon dou­bled about every seven months. Used in Chap­ters 6, 10.

Part VII. Sys­tems at scale

• ZeRO: Memory Opti­miza­tions Toward Train­ing Tril­lion Param­e­ter Models (2019) [22]. ZeRO: shard opti­mizer states, gra­di­ents and weights across data-par­al­lel ranks to cut memory per GPU. Used in Chap­ters 1, 11, 15.
• Effi­cient Large-Scale Lan­guage Model Train­ing on GPU Clus­ters Using Mega­tron-LM (2021) [41]. Mega­tron-LM at scale: tensor, pipe­line and data par­al­lel­ism com­posed, with the inter­leaved pipe­line sched­ule. Used in Chap­ters 1, 2, 11, 15.
• Reduc­ing Acti­va­tion Recom­pu­ta­tion in Large Trans­former Models (2022) [45]. Sequence par­al­lel­ism and selec­tive recom­pu­ta­tion, with the acti­va­tion-memory for­mu­las this guide uses. Used in Chap­ters 1, 11, 15.
• MegaScale: Scal­ing Large Lan­guage Model Train­ing to More Than 10,000 GPUs (2024) [47]. Train­ing on more than 10,000 GPUs in pro­duc­tion: com­mu­ni­ca­tion over­lap, diag­nos­tics and fault tol­er­ance. Used in Chap­ters 1, 2, 11, 12, 15.
• Revis­it­ing Reli­a­bil­ity in Large-Scale Machine Learn­ing Research Clus­ters (2024) [903]. Fail­ure sta­tis­tics from two large research clus­ters and a model of effec­tive train­ing time. Used in Chap­ters 2, 12, 15.

Part VIII. Deploy­ment

• Effi­ciently Scal­ing Trans­former Infer­ence (2022) [33]. The ana­lyt­i­cal model of trans­former infer­ence: pre­fill against decode, memory-bound latency and how to par­ti­tion a model. Used in Chap­ters 1, 13, 15.
• Effi­cient Memory Man­age­ment for Large Lan­guage Model Serv­ing with PagedAttention (2023) [73]. PagedAttention and vLLM: KV-cache memory man­aged in blocks, which mul­ti­plies how many requests fit on a GPU. Used in Chap­ters 1, 13.
• Fast Infer­ence from Trans­form­ers via Spec­u­la­tive Decod­ing (2022) [1017]. Spec­u­la­tive decod­ing: a small draft model pro­poses tokens that the large model checks in one pass, with unchanged out­puts. Used in Chap­ters 13, 15.
• SWE-bench: Can Lan­guage Models Resolve Real-World GitHub Issues? (2023) [814]. SWE-bench: real GitHub issues graded by the repos­i­tory's own tests inside a con­tainer. Used in Chap­ters 10, 14.
• Fire­cracker: Light­weight Vir­tu­al­iza­tion for Server­less Appli­ca­tions (2020) [1087]. The microVM design behind fast, strongly iso­lated sand­boxes. Used in Chap­ter 14.

Part IX. Prac­tice

• A Recipe for Train­ing Neural Net­works (2019) [943]. A prac­ti­tioner's check­list for train­ing neural net­works: start simple, check every step and expect silent bugs. Used in Chap­ters 12, 15.
• The Ultra-Scale Play­book: Train­ing LLMs on GPU Clus­ters (2025) [44]. An inter­ac­tive, bench­marked guide to every par­al­lel­ism strat­egy, from one GPU to thou­sands. Used in Chap­ters 1, 11, 15.
• How To Scale Your Model (2025) [1145]. A short text­book on scal­ing models across accel­er­a­tors, with the roofline and com­mu­ni­ca­tion arith­metic worked out. Used in Chap­ter 15.
• The Smol Train­ing Play­book: The Secrets to Build­ing World-Class LLMs (2025) [13]. A candid account of build­ing SmolLM3, includ­ing its abla­tions, fail­ures and the com­pute they con­sumed. Used in Chap­ters 3, 4, 5, 6, 7, 15.

Notes

1. The first ver­sion of the R1 report, from Jan­u­ary 2025, gave 71.0% for an ear­lier check­point. This guide uses the later ver­sion's 77.9%. ↩
2. Plug­ging Table 6 into $C=6ND$ gives $1.8C_{min}$ at every budget, because $D_{\text{opt}}$ assumes train­ing at the crit­i­cal batch size, where $C=2C_{min}$ (com­puted) [25, 360] ↩
3. The fits add an offset, $1-e^{-P/\gamma +n}$. With the pub­lished con­stants (${\gamma }_w=2.6745$, $n_w=0.3037$, ${\gamma }_a=2.2102$, $n_a=1.4072$, ${\gamma }_{kv}=0.9578$, $n_{kv}=2.4185$), max­i­miz­ing $N_{\text{eff}}/P$ gives 7.0 bits (com­puted) [458] ↩

References

1. Cheng­gang Zhao, Chengqi Deng, Chong Ruan et al. (2025). Insights into DeepSeek-V3: Scal­ing Chal­lenges and Reflec­tions on Hard­ware for AI Archi­tec­tures. arXiv:2505.09343.
2. DeepSeek-AI, Aixin Liu, Bei Feng et al. (2024). DeepSeek-V3 Tech­ni­cal Report. arXiv:2412.19437.
3. DeepSeek-AI, Daya Guo, Dejian Yang et al. (2025). DeepSeek-R1: Incen­tiviz­ing Rea­son­ing Capa­bil­ity in LLMs via Rein­force­ment Learn­ing. arXiv:2501.12948.
4. NVIDIA (2024). H100 GPU. NVIDIA prod­uct page and spec­i­fi­ca­tions.
5. Guil­herme Penedo, Hynek Kydlíček, Loubna Ben allal et al. (2024). The FineWeb Datasets: Decant­ing the Web for the Finest Text Data at Scale. arXiv:2406.17557.
6. Aaron Grattafiori, Abhi­manyu Dubey, Abhi­nav Jauhri et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
7. Team OLMo, Pete Walsh, Luca Sol­daini et al. (2024). 2 OLMo 2 Furi­ous. arXiv:2501.00656.
8. Long Ouyang, Jeff Wu, Xu Jiang et al. (2022). Train­ing lan­guage models to follow instruc­tions with human feed­back. arXiv:2203.02155.
9. Michael Luo, Naman Jain, Jaski­rat Singh et al. (2025). DeepSWE: Train­ing a Fully Open-sourced, State-of-the-Art Coding Agent by Scal­ing RL. Blog post (Agen­tica and Together AI).
10. Jialiang Huang, Hongx­uan Tang, Jingchang Chen et al. (2026). DeepSeek Elas­tic Com­pute (DSec): A Sand­box Infra­struc­ture for Effec­tive Agen­tic Train­ing at Scale. arXiv:2609.22978.
11. Meta (2024). Llama 3.1 Model Card. GitHub, meta-llama/llama-models.
12. Team Olmo, Allyson Ettinger, Amanda Bertsch et al. (2025). Olmo 3. arXiv:2512.13961.
13. Loubna Ben Allal, Lewis Tun­stall, Noua­mane Tazi et al. (2025). The Smol Train­ing Play­book: The Secrets to Build­ing World-Class LLMs. Hug­ging Face.
14. DeepSeek-AI (2025). Day 6: One More Thing, DeepSeek-V3/R1 Infer­ence System Over­view. GitHub, deepseek-ai/open-infra-index.
15. Andrej Karpa­thy (2024). Repro­duc­ing GPT-2 (124M) in llm.c in 90 min­utes for $20. GitHub dis­cus­sion, karpa­thy/llm.c.
16. Neil Chowd­hury, James Aung, Chan Jun Shern et al. (2024). Intro­duc­ing SWE-bench Ver­i­fied. OpenAI blog (archived copy).
17. Unsloth AI (2024). config.json · unsloth/Meta-Llama-3.1⁠–⁠8B. Hug­ging Face model repos­i­tory (public mirror of Meta Llama 3.1 8B config).
18. DeepSeek-AI (2024). DeepSeek-V3 config.json. Hug­ging Face model repos­i­tory.
19. Alec Rad­ford, Jef­frey Wu, Rewon Child et al. (2019). Lan­guage Models are Unsu­per­vised Mul­ti­task Learn­ers. OpenAI tech­ni­cal report.
20. OpenAI (2019). config.json · openai-com­mu­nity/gpt2. Hug­ging Face model repos­i­tory.
21. Allen Insti­tute for AI (2024). config.json · allenai/OLMo-2⁠–⁠1124⁠–⁠7B. Hug­ging Face model repos­i­tory.
22. Samyam Rajb­han­dari, Jeff Rasley, Olatunji Ruwase and Yux­iong He (2019). ZeRO: Memory Opti­miza­tions Toward Train­ing Tril­lion Param­e­ter Models. arXiv:1910.02054.
23. Quentin Anthony, Stella Bider­man and Hailey Schoelkopf (2023). Trans­former Math 101. EleutherAI Blog.
24. Meta Llama (2024). llama3/llama/tok­enizer.py at main · meta-llama/llama3. GitHub repos­i­tory (source file).
25. Jared Kaplan, Sam McCandlish, Tom Henighan et al. (2020). Scal­ing Laws for Neural Lan­guage Models. arXiv:2001.08361.
26. Noam Shazeer (2020). GLU Vari­ants Improve Trans­former. arXiv:2002.05202.
27. Tomer Porian, Mitchell Worts­man, Jenia Jitsev, Ludwig Schmidt and Yair Carmon (2024). Resolv­ing Dis­crep­an­cies in Com­pute-Opti­mal Scal­ing of Lan­guage Models. arXiv:2406.19146.
28. OpenAI, Sand­hini Agar­wal, Lama Ahmad et al. (2025). gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925.
29. Quentin Anthony, Jacob Hatef, Deepak Narayanan et al. (2024). The Case for Co-Design­ing Model Archi­tec­tures with Hard­ware. arXiv:2401.14489.
30. William Fedus, Barret Zoph and Noam Shazeer (2021). Switch Trans­form­ers: Scal­ing to Tril­lion Param­e­ter Models with Simple and Effi­cient Spar­sity. arXiv:2101.03961.
31. DeepSeek-AI (2024). deepseek-ai/DeepSeek-V3 · Hug­ging Face. Hug­ging Face model card.
32. NVIDIA (2023). Matrix Mul­ti­pli­ca­tion Back­ground User's Guide. NVIDIA Deep Learn­ing Per­for­mance Doc­u­men­ta­tion.
33. Reiner Pope, Sholto Dou­glas, Aakanksha Chowd­h­ery et al. (2022). Effi­ciently Scal­ing Trans­former Infer­ence. arXiv:2211.05102.
34. Aakanksha Chowd­h­ery, Sharan Narang, Jacob Devlin et al. (2022). PaLM: Scal­ing Lan­guage Mod­el­ing with Path­ways. arXiv:2204.02311.
35. Jaime Sevilla, Lennart Heim, Marius Hobb­hahn et al. (2022). Esti­mat­ing train­ing com­pute of deep learn­ing models. Epoch AI report.
36. Tom B. Brown, Ben­jamin Mann, Nick Ryder et al. (2020). Lan­guage Models are Few-Shot Learn­ers. arXiv:2005.14165.
37. Andrej Karpa­thy (2023). nanoGPT/model.py at master · karpa­thy/nanoGPT. GitHub repos­i­tory (source file).
38. Tri Dao (2023). FlashAttention-2: Faster Atten­tion with Better Par­al­lel­ism and Work Par­ti­tion­ing. arXiv:2307.08691.
39. Jacob Austin, Sholto Dou­glas, Roy Frostig et al. (2025). All the Trans­former Math You Need to Know. How To Scale Your Model (online book), Part 4.
40. Jordan Hoff­mann, Sebas­tian Borgeaud, Arthur Mensch et al. (2022). Train­ing Com­pute-Opti­mal Large Lan­guage Models. arXiv:2203.15556.
41. Deepak Narayanan, Moham­mad Shoeybi, Jared Casper et al. (2021). Effi­cient Large-Scale Lan­guage Model Train­ing on GPU Clus­ters Using Mega­tron-LM. arXiv:2104.04473.
42. DeepSeek-AI, Xiao Bi, Deli Chen et al. (2024). DeepSeek LLM: Scal­ing Open-Source Lan­guage Models with Longter­mism. arXiv:2401.02954.
43. Tim Pearce and Jinyeop Song (2024). Rec­on­cil­ing Kaplan and Chin­chilla Scal­ing Laws. arXiv:2406.12907.
44. Noua­mane Tazi, Fer­di­nand Mom, Haojun Zhao et al. (2025). The Ultra-Scale Play­book: Train­ing LLMs on GPU Clus­ters. Hug­ging Face.
45. Vijay Kor­thikanti, Jared Casper, Sangkug Lym et al. (2022). Reduc­ing Acti­va­tion Recom­pu­ta­tion in Large Trans­former Models. arXiv:2205.05198.
46. Hugo Tou­vron, Thibaut Lavril, Gau­tier Izac­ard et al. (2023). LLaMA: Open and Effi­cient Foun­da­tion Lan­guage Models. arXiv:2302.13971.
47. Ziheng Jiang, Haibin Lin, Yinmin Zhong et al. (2024). MegaScale: Scal­ing Large Lan­guage Model Train­ing to More Than 10,000 GPUs. arXiv:2402.15627.
48. Wan­chao Liang, Tianyu Liu, Less Wright et al. (2024). TorchTitan: One-stop PyTorch native solu­tion for pro­duc­tion ready LLM pre-train­ing. arXiv:2410.06511.
49. Dennis Liu, Zijie Yan, Xin Yao et al. (2025). MoE Par­al­lel Fold­ing: Het­ero­ge­neous Par­al­lel­ism Map­pings for Effi­cient Large-Scale MoE Model Train­ing with Mega­tron Core. arXiv:2504.14960.
50. Chao Jin, Ziheng Jiang, Zhihao Bai et al. (2025). MegaScale-MoE: Large-Scale Com­mu­ni­ca­tion-Effi­cient Train­ing of Mix­ture-of-Experts Models in Pro­duc­tion. arXiv:2505.11432.
51. Paulius Micike­vi­cius, Sharan Narang, Jonah Alben et al. (2017). Mixed Pre­ci­sion Train­ing. arXiv:1710.03740.
52. Dhiraj Kalamkar, Dhee­vatsa Mudi­gere, Naveen Mellem­pudi et al. (2019). A Study of BFLOAT16 for Deep Learn­ing Train­ing. arXiv:1905.12322.
53. Tim Dettmers, Mike Lewis, Sam Shleifer and Luke Zettle­moyer (2021). 8-bit Opti­miz­ers via Block-wise Quan­ti­za­tion. arXiv:2110.02861.
54. Noam Shazeer and Mitchell Stern (2018). Adafac­tor: Adap­tive Learn­ing Rates with Sub­lin­ear Memory Cost. arXiv:1804.04235.
55. Bin­grui Li, Jian­fei Chen and Jun Zhu (2023). Memory Effi­cient Opti­miz­ers with 4-bit States. arXiv:2309.01507.
56. Jiawei Zhao, Zhenyu Zhang, Beidi Chen et al. (2024). GaLore: Memory-Effi­cient LLM Train­ing by Gra­di­ent Low-Rank Pro­jec­tion. arXiv:2403.03507.
57. Houwen Peng, Kan Wu, Yixuan Wei et al. (2023). FP8-LM: Train­ing FP8 Large Lan­guage Models. arXiv:2310.18313.
58. Haocheng Xi, Han Cai, Ligeng Zhu et al. (2024). COAT: Com­press­ing Opti­mizer states and Acti­va­tion for Memory-Effi­cient FP8 Train­ing. arXiv:2410.19313.
59. Mahdi Nikdan, Amir Zandieh, Dan Alis­tarh and Vahab Mir­rokni (2026). ECO: Quan­tized Train­ing with­out Full-Pre­ci­sion Master Weights. arXiv:2601.22101.
60. Kristi Topol­lai and Anna Choro­man­ska (2026). Under­stand­ing Quan­ti­za­tion of Opti­mizer States in LLM Pre-train­ing: Dynam­ics of State Stal­e­ness and Effec­tive­ness of State Resets. arXiv:2603.16731.
61. Jie Ren, Samyam Rajb­han­dari, Reza Yaz­dani Aminabadi et al. (2021). ZeRO-Off­load: Democ­ra­tiz­ing Bil­lion-Scale Model Train­ing. arXiv:2101.06840.
62. Samyam Rajb­han­dari, Olatunji Ruwase, Jeff Rasley, Shaden Smith and Yux­iong He (2021). ZeRO-Infin­ity: Break­ing the GPU Memory Wall for Extreme Scale Deep Learn­ing. arXiv:2104.07857.
63. Tri Dao, Daniel Y. Fu, Ste­fano Ermon, Atri Rudra and Christo­pher Ré (2022). FlashAttention: Fast and Memory-Effi­cient Exact Atten­tion with IO-Aware­ness. arXiv:2205.14135.
64. Tianqi Chen, Bing Xu, Chiyuan Zhang and Carlos Guestrin (2016). Train­ing Deep Nets with Sub­lin­ear Memory Cost. arXiv:1604.06174.
65. Erik Wij­mans, Brody Huval, Alexan­der Hertzberg, Vladlen Koltun and Philipp Krähenbühl (2024). Cut Your Losses in Large-Vocab­u­lary Lan­guage Models. arXiv:2411.09009.
66. Pin-Lun Hsu, Yun Dai, Vig­nesh Kotha­palli et al. (2024). Liger Kernel: Effi­cient Triton Ker­nels for LLM Train­ing. arXiv:2410.10989.
67. Kazuki Fujii, Kohei Watan­abe and Rio Yokota (2024). Accel­er­at­ing Large Lan­guage Model Train­ing with 4D Par­al­lel­ism and Memory Con­sump­tion Esti­ma­tor. arXiv:2411.06465.
68. PyTorch Con­trib­u­tors (2025). CUDA seman­tics. PyTorch 2.9 doc­u­men­ta­tion.
69. PyTorch Con­trib­u­tors (2025). Under­stand­ing CUDA Memory Usage. PyTorch 2.9 doc­u­men­ta­tion.
70. Nikhil Sar­dana, Jacob Portes, Sasha Doubov and Jonathan Fran­kle (2023). Beyond Chin­chilla-Opti­mal: Account­ing for Infer­ence in Lan­guage Model Scal­ing Laws. arXiv:2401.00448.
71. Char­lie Snell, Jae­hoon Lee, Kelvin Xu and Aviral Kumar (2024). Scal­ing LLM Test-Time Com­pute Opti­mally can be More Effec­tive than Scal­ing Model Param­e­ters. arXiv:2408.03314.
72. Kimi Team, Yifan Bai, Yiping Bao et al. (2025). Kimi K2: Open Agen­tic Intel­li­gence. arXiv:2507.20534.
73. Woosuk Kwon, Zhuo­han Li, Siyuan Zhuang et al. (2023). Effi­cient Memory Man­age­ment for Large Lan­guage Model Serv­ing with PagedAttention. arXiv:2309.06180.
74. Noam Shazeer (2019). Fast Trans­former Decod­ing: One Write-Head is All You Need. arXiv:1911.02150.
75. Joshua Ainslie, James Lee-Thorp, Michiel de Jong et al. (2023). GQA: Train­ing Gen­er­al­ized Multi-Query Trans­former Models from Multi-Head Check­points. arXiv:2305.13245.
76. Tim Dettmers, Artidoro Pagnoni, Ari Holtz­man and Luke Zettle­moyer (2023). QLoRA: Effi­cient Fine­tun­ing of Quan­tized LLMs. arXiv:2305.14314.
77. Horace He (2022). Making Deep Learn­ing go Brrrr From First Prin­ci­ples. Blog post.
78. Samuel Williams, Andrew Water­man and David Pat­ter­son (2009). Roofline: an insight­ful visual per­for­mance model for mul­ti­core archi­tec­tures. Com­mu­ni­ca­tions of the ACM.
79. Samuel Webb Williams, Andrew Water­man and David A. Pat­ter­son (2008). Roofline: An Insight­ful Visual Per­for­mance Model for Float­ing-Point Pro­grams and Mul­ti­core Archi­tec­tures. Tech­ni­cal Report UCB/EECS-2008⁠–⁠134, Uni­ver­sity of Cal­i­for­nia at Berke­ley.
80. NVIDIA (2023). GPU Per­for­mance Back­ground User's Guide. NVIDIA Deep Learn­ing Per­for­mance Doc­u­men­ta­tion.
81. Weile Luo, Ruibo Fan, Zeyu Li et al. (2024). Bench­mark­ing and Dis­sect­ing the Nvidia Hopper GPU Archi­tec­ture. arXiv:2402.13499.
82. Jay Shah, Ganesh Bik­shandi, Ying Zhang et al. (2024). FlashAttention-3: Fast and Accu­rate Atten­tion with Asyn­chrony and Low-pre­ci­sion. arXiv:2407.08608.
83. NVIDIA (2020). NVIDIA A100. NVIDIA prod­uct page and spec­i­fi­ca­tions.
84. NVIDIA (2024). H200 GPU. NVIDIA prod­uct page and spec­i­fi­ca­tions.
85. NVIDIA (2026). NVIDIA HGX Plat­form. NVIDIA prod­uct page and spec­i­fi­ca­tions.
86. NVIDIA (2024). DGX B200. NVIDIA prod­uct page and spec­i­fi­ca­tions.
87. AMD (2023). AMD Instinct MI300X Accel­er­a­tors. AMD prod­uct page and spec­i­fi­ca­tions.
88. Google Cloud (2026). TPU v5p | Google Cloud Doc­u­men­ta­tion. Google Cloud doc­u­men­ta­tion.
89. Amir Gho­lami, Zhewei Yao, Sehoon Kim et al. (2024). AI and Memory Wall. arXiv:2403.14123.
90. Zhi­hang Yuan, Yuzhang Shang, Yang Zhou et al. (2024). LLM Infer­ence Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363.
91. Jacob Austin, Sholto Dou­glas, Roy Frostig et al. (2025). All About Trans­former Infer­ence. How To Scale Your Model (online book), Part 7.
92. Pratyush Patel, Esha Choukse, Chao­jie Zhang et al. (2023). Split­wise: Effi­cient gen­er­a­tive LLM infer­ence using phase split­ting. arXiv:2311.18677.
93. David Pat­ter­son, Joseph Gon­za­lez, Quoc Le et al. (2021). Carbon Emis­sions and Large Neural Net­work Train­ing. arXiv:2104.10350.
94. Cooper Elsworth, Keguo Huang, David Pat­ter­son et al. (2025). Mea­sur­ing the envi­ron­men­tal impact of deliv­er­ing AI at Google Scale. arXiv:2508.15734.
95. Jacob Mor­ri­son, Clara Na, Jared Fer­nan­dez et al. (2025). Holis­ti­cally Eval­u­at­ing the Envi­ron­men­tal Impact of Cre­at­ing Lan­guage Models. arXiv:2503.05804.
96. Jae-Won Chung, Jeff J. Ma, Ruofan Wu et al. (2025). The ML.ENERGY Bench­mark: Toward Auto­mated Infer­ence Energy Mea­sure­ment and Opti­miza­tion. arXiv:2505.06371.
97. Alexan­dra Sasha Luc­cioni, Syl­vain Vigu­ier and Anne-Laure Ligozat (2022). Esti­mat­ing the Carbon Foot­print of BLOOM, a 176B Param­e­ter Lan­guage Model. arXiv:2211.02001.
98. Epoch AI (2026). Data on AI models. Epoch AI dataset (notable_ai_models.csv, accessed 2026-11-03).
99. Lambda (2026). GPU cloud pric­ing: rent NVIDIA H100, H200, and B200. Vendor pric­ing page (accessed 2026-11-03).
100. Ben Cot­tier, Robi Rahman, Loredana Fat­torini et al. (2024). The rising costs of train­ing fron­tier AI models. arXiv:2405.21015.
101. Jaime Sevilla and Edu Roldán (2024). Train­ing com­pute of fron­tier AI models grows by 4⁠–⁠5x per year. Epoch AI report.
102. Anson Ho, Tamay Besiroglu, Ege Erdil et al. (2024). Algo­rith­mic progress in lan­guage models. arXiv:2403.05812.
103. Jaime Sevilla, Tamay Besiroglu, Ben Cot­tier et al. (2024). Can AI scal­ing con­tinue through 2030?. Epoch AI report.
104. Michael Ander­sch, Greg Palmer, Ronny Krashin­sky et al. (2022). NVIDIA Hopper Archi­tec­ture In-Depth. NVIDIA Tech­ni­cal Blog.
105. Ben­jamin Spec­tor, Aaryan Sing­hal, Simran Arora and Chris Re (2024). GPUs Go Brrr. Hazy Research blog.
106. NVIDIA (2026). Pro­gram­ming Model - CUDA Pro­gram­ming Guide. CUDA Pro­gram­ming Guide.
107. NVIDIA (2026). NVIDIA Hopper Tuning Guide. CUDA Tool­kit doc­u­men­ta­tion.
108. NVIDIA (2017). NVIDIA Tesla V100 GPU Archi­tec­ture. NVIDIA whitepa­per WP-08608⁠–⁠001_v1.1.
109. NVIDIA (2020). NVIDIA A100 Tensor Core GPU Archi­tec­ture. NVIDIA whitepa­per.
110. Ted Zadouri, Markus Hoehner­bach, Jay Shah et al. (2026). FlashAttention-4: Algo­rithm and Kernel Pipe­lin­ing Co-Design for Asym­met­ric Hard­ware Scal­ing. arXiv:2603.05451.
111. Google Cloud (2026). TPU archi­tec­ture. Google Cloud doc­u­men­ta­tion.
112. Norman P. Jouppi, George Kurian, Sheng Li et al. (2023). TPU v4: An Opti­cally Recon­fig­urable Super­com­puter for Machine Learn­ing with Hard­ware Sup­port for Embed­dings. arXiv:2304.01433.
113. AMD (2023). AMD Instinct MI300X Accel­er­a­tor. AMD data sheet.
114. AMD (2025). AMD Instinct MI355X GPUs. AMD prod­uct page.
115. AMD (2023). AMD CDNA 3 Archi­tec­ture. AMD whitepa­per.
116. Bita Darvish Rouhani, Nitin Gare­grat, Tom Savell et al. (2023). OCP Microscal­ing For­mats (MX) Spec­i­fi­ca­tion Ver­sion 1.0. Open Com­pute Project.
117. Paulius Micike­vi­cius, Dusan Stosic, Neil Burgess et al. (2022). FP8 For­mats for Deep Learn­ing. arXiv:2209.05433.
118. Norman P. Jouppi, Srid­har Lak­sh­mana­murthy, Cliff Young and David Pat­ter­son (2026). Google's Train­ing Super­com­put­ers from TPU v2 to Iron­wood: Archi­tec­tural Sta­bil­ity, Scale, Resilience, Power Effi­ciency, and Sus­tain­abil­ity Across Five Gen­er­a­tions. arXiv:2606.15870.
119. Asit Mishra, Dusan Stosic, Simon Layton and Paulius Micike­vi­cius (2025). Recipes for Pre-train­ing LLMs with MXFP8. arXiv:2506.08027.
120. Eduardo Alvarez, Omri Almog, Eric Chung et al. (2025). Intro­duc­ing NVFP4 for Effi­cient and Accu­rate Low-Pre­ci­sion Infer­ence. NVIDIA Tech­ni­cal Blog.
121. NVIDIA, Felix Abecas­sis, Anjulie Agrusa et al. (2025). Pre­train­ing Large Lan­guage Models with NVFP4. arXiv:2509.25149.
122. Brian Chmiel, Maxim Fish­man, Ron Banner and Daniel Soudry (2025). FP4 All the Way: Fully Quan­tized Train­ing of LLMs. arXiv:2505.19115.
123. Albert Tseng, Tao Yu and Young­suk Park (2025). Train­ing LLMs with MXFP4. arXiv:2502.20586.
124. Roberto L. Castro, Andrei Pan­ferov, Soroush Tabesh et al. (2025). Quar­tet: Native FP4 Train­ing Can Be Opti­mal for Large Lan­guage Models. arXiv:2505.14669.
125. Yubo Wang, Xueguang Ma, Ge Zhang et al. (2024). MMLU-Pro: A More Robust and Chal­leng­ing Multi-Task Lan­guage Under­stand­ing Bench­mark. arXiv:2406.01574.
126. Maxim Fish­man, Brian Chmiel, Ron Banner and Daniel Soudry (2024). Scal­ing FP8 train­ing to tril­lion-token LLMs. arXiv:2409.12517.
127. Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shi­gang Li and Torsten Hoe­fler (2020). Data Move­ment Is All You Need: A Case Study on Opti­miz­ing Trans­form­ers. arXiv:2007.00072.
128. Maxim Milakov and Natalia Gimelshein (2018). Online nor­mal­izer cal­cu­la­tion for soft­max. arXiv:1805.02867.
129. Ben­jamin F. Spec­tor, Simran Arora, Aaryan Sing­hal, Daniel Y. Fu and Christo­pher Ré (2024). ThunderKittens: Simple, Fast, and Adorable AI Ker­nels. arXiv:2410.20399.
130. Quentin Anthony, Yury Tok­panov, Skyler Szot et al. (2025). Train­ing Foun­da­tion Models on a Full-Stack AMD Plat­form: Com­pute, Net­work­ing, and System Design. arXiv:2511.17127.
131. Juechu Dong, Boyuan Feng, Driss Gues­sous, Yanbo Liang and Horace He (2024). Flex Atten­tion: A Pro­gram­ming Model for Gen­er­at­ing Opti­mized Atten­tion Ker­nels. arXiv:2412.05496.
132. Philippe Tillet, H. T. Kung and David Cox (2019). Triton: an inter­me­di­ate lan­guage and com­piler for tiled neural net­work com­pu­ta­tions. Pro­ceed­ings of the 3rd ACM SIGPLAN Inter­na­tional Work­shop on Machine Learn­ing and Pro­gram­ming Lan­guages.
133. Jason Ansel, Edward Yang, Horace He et al. (2024). PyTorch 2: Faster Machine Learn­ing Through Dynamic Python Byte­code Trans­for­ma­tion and Graph Com­pi­la­tion. Pro­ceed­ings of the 29th ACM Inter­na­tional Con­fer­ence on Archi­tec­tural Sup­port for Pro­gram­ming Lan­guages and Oper­at­ing Sys­tems, Volume 2.
134. Alan Gray (2019). Get­ting Started with CUDA Graphs. NVIDIA Tech­ni­cal Blog.
135. NVIDIA (2025). NVIDIA/cut­lass: CUDA Tem­plates and Python DSLs for High-Per­for­mance Linear Alge­bra. GitHub repos­i­tory.
136. Lei Wang, Yu Cheng, Yining Shi et al. (2025). TileLang: A Com­pos­able Tiled Pro­gram­ming Model for AI Sys­tems. arXiv:2504.17577.
137. NVIDIA (2026). Intro­duc­tion to NVIDIA DGX H100/H200 Sys­tems. NVIDIA DGX H100/H200 User Guide.
138. Jacob Austin, Sholto Dou­glas, Roy Frostig et al. (2025). How to Think About GPUs. How To Scale Your Model (online book).
139. NVIDIA (2025). GB200 NVL72. NVIDIA prod­uct page.
140. NVIDIA (2026). NVLink & NVLink Switch for Advanced Multi-GPU Com­mu­ni­ca­tion. NVIDIA prod­uct page.
141. Adithya Gangidi, Rui Miao, Sheng­bao Zheng et al. (2024). RDMA over Eth­er­net for Dis­trib­uted Train­ing at Meta Scale. Pro­ceed­ings of the ACM SIGCOMM 2024 Con­fer­ence.
142. NVIDIA (2026). 1. Over­view — GPUDirect RDMA 13.4 doc­u­men­ta­tion. CUDA doc­u­men­ta­tion.
143. NVIDIA (2024). Per­for­mance reported by NCCL tests. GitHub (NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md).
144. Chuanx­iong Guo, Haitao Wu, Zhong Deng et al. (2016). RDMA over Com­mod­ity Eth­er­net at Scale. Pro­ceed­ings of the 2016 ACM SIGCOMM Con­fer­ence.
145. Kevin Lee, Adi Gangidi and Mathew Oldham (2024). Build­ing Meta’s GenAI Infra­struc­ture. Engi­neer­ing at Meta blog.
146. Moham­mad Al-Fares, Alexan­der Loukissas and Amin Vahdat (2008). A scal­able, com­mod­ity data center net­work archi­tec­ture. Pro­ceed­ings of the ACM SIGCOMM 2008 con­fer­ence on Data com­mu­ni­ca­tion.
147. Joao Araujo, Alex Chow, Mark Han­d­ley et al. (2026). Resilient AI Super­com­puter Net­work­ing using MRC and SRv6. arXiv:2605.04333.
148. Min Si, Pavan Balaji, Yongzhou Chen et al. (2025). Col­lec­tive Com­mu­ni­ca­tion for 100k+ GPUs. arXiv:2510.20171.
149. Kun Qian, Yongqing Xi, Jiamin Cao et al. (2024). Alibaba HPN: A Data Center Net­work for Large Lan­guage Model Train­ing. Pro­ceed­ings of the ACM SIGCOMM 2024 Con­fer­ence.
150. xAI (2025). Colos­sus: The World's Largest AI Super­com­puter. Com­pany page (x.ai; footer '© 2026 SpaceX Super Intel­li­gence LLC', title suffix 'SpaceXSI' as of 2026⁠–⁠11).
151. NVIDIA (2026). Hard­ware - NVIDIA DGX GB Rack Scale Sys­tems User Guide. NVIDIA doc­u­men­ta­tion.
152. Amin Vahdat (2025). Iron­wood: The first Google TPU for the age of infer­ence. Google blog.
153. Kon­stantin F. Pilz, James Sanders, Robi Rahman and Lennart Heim (2025). Trends in AI Super­com­put­ers. arXiv:2504.16026.
154. Google (2026). Power usage effec­tive­ness - Google Data Cen­ters. Google data cen­ters page.
155. Esha Choukse, Bri­jesh War­rier, Scot Heath et al. (2025). Power Sta­bi­liza­tion for AI Train­ing Dat­a­cen­ters. arXiv:2508.14318.
156. Google Cloud (2026). TPU7x (Iron­wood) | Google Cloud Doc­u­men­ta­tion. Google Cloud doc­u­men­ta­tion.
157. Alicia Golden, Carole-Jean Wu, Gu-Yeon Wei and David Brooks (2026). The xPU-athalon: Quan­ti­fy­ing the Com­pe­ti­tion of AI Accel­er­a­tion. arXiv:2604.10852.
158. Epoch AI (2026). Data on machine learn­ing hard­ware. Epoch AI dataset.
159. AMD (2024). AMD Instinct MI325X Accel­er­a­tors. AMD prod­uct page.
160. Google Cloud (2026). TPU v5e | Google Cloud Doc­u­men­ta­tion. Google Cloud doc­u­men­ta­tion.
161. Google Cloud (2026). TPU v6e | Google Cloud Doc­u­men­ta­tion. Google Cloud doc­u­men­ta­tion.
162. AWS (2025). Trainium2 Archi­tec­ture - AWS Neuron Doc­u­men­ta­tion. AWS Neuron doc­u­men­ta­tion.
163. AWS (2025). Amazon EC2 Trn2 Archi­tec­ture - AWS Neuron Doc­u­men­ta­tion. AWS Neuron doc­u­men­ta­tion.
164. Intel (2025). Intel Gaudi 3 AI Accel­er­a­tor Tech­ni­cal Paper. Intel white paper.
165. Cere­bras Sys­tems (2024). Cere­bras Sys­tems Unveils World's Fastest AI Chip with Whop­ping 4 Tril­lion Tran­sis­tors. Press release.
166. Guil­herme Penedo, Hynek Kydlicek, Loubna Ben Allal et al. (2024). FineWeb: decant­ing the web for the finest text data at scale. Hug­ging Face blog.
167. Jef­frey Li, Alex Fang, Geor­gios Smyr­nis et al. (2024). DataComp-LM: In search of the next gen­er­a­tion of train­ing sets for lan­guage models. arXiv:2406.11794.
168. Guil­herme Penedo, Quentin Malar­tic, Daniel Hess­low et al. (2023). The RefinedWeb Dataset for Falcon LLM: Out­per­form­ing Curated Cor­pora with Web Data, and Web Data Only. arXiv:2306.01116.
169. Anton Lozhkov, Loubna Ben Allal, Lean­dro von Werra and Thomas Wolf (2024). FineWeb-Edu: the Finest Col­lec­tion of Edu­ca­tional Con­tent. Dataset card.
170. Common Crawl Foun­da­tion (2025). Get Started - Common Crawl. Common Crawl doc­u­men­ta­tion.
171. Adrien Bar­baresi (2021). Trafi­latura: A Web Scrap­ing Library and Com­mand-Line Tool for Text Dis­cov­ery and Extrac­tion. Pro­ceed­ings of the 59th Annual Meet­ing of the Asso­ci­a­tion for Com­pu­ta­tional Lin­guis­tics and the 11th Inter­na­tional Joint Con­fer­ence on Nat­u­ral Lan­guage Pro­cess­ing: System Demon­stra­tions.
172. Janek Beven­dorff (2024). Resili­parse doc­u­men­ta­tion. Soft­ware doc­u­men­ta­tion.
173. Jef­frey Li, Josh Gard­ner, Doug Kang et al. (2026). Beyond a Single Extrac­tor: Re-think­ing HTML-to-Text Extrac­tion for LLM Pre­train­ing. arXiv:2602.19548.
174. Dan Su, Kezhi Kong, Ying Lin et al. (2024). Nemotron-CC: Trans­form­ing Common Crawl into a Refined Long-Hori­zon Pre­train­ing Dataset. arXiv:2412.02595.
175. Armand Joulin, Edouard Grave, Piotr Bojanowski and Tomas Mikolov (2016). Bag of Tricks for Effi­cient Text Clas­si­fi­ca­tion. arXiv:1607.01759.
176. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin and Tomas Mikolov (2018). Learn­ing Word Vec­tors for 157 Lan­guages. arXiv:1802.06893.
177. Colin Raffel, Noam Shazeer, Adam Roberts et al. (2019). Explor­ing the Limits of Trans­fer Learn­ing with a Uni­fied Text-to-Text Trans­former. arXiv:1910.10683.
178. Luca Sol­daini, Rodney Kinney, Akshita Bhagia et al. (2024). Dolma: an Open Corpus of Three Tril­lion Tokens for Lan­guage Model Pre­train­ing Research. arXiv:2402.00159.
179. Guil­herme Penedo, Hynek Kydlíček, Vinko Sabolčec et al. (2025). FineWeb2: One Pipe­line to Scale Them All -- Adapt­ing Pre-Train­ing Data Pro­cess­ing to Every Lan­guage. arXiv:2506.20920.
180. Amir Hos­sein Kar­garan, Ayyoob Imani, François Yvon and Hin­rich Schütze (2023). GlotLID: Lan­guage Iden­ti­fi­ca­tion for Low-Resource Lan­guages. arXiv:2310.16248.
181. Jesse Dodge, Maarten Sap, Ana Marasović et al. (2021). Doc­u­ment­ing Large Web­text Cor­pora: A Case Study on the Colos­sal Clean Crawled Corpus. arXiv:2104.08758.
182. Jack W. Rae, Sebas­tian Borgeaud, Trevor Cai et al. (2021). Scal­ing Lan­guage Models: Meth­ods, Anal­y­sis & Insights from Train­ing Gopher. arXiv:2112.11446.
183. Johannes Welbl, Amelia Glaese, Jonathan Uesato et al. (2021). Chal­lenges in Detox­i­fy­ing Lan­guage Models. arXiv:2109.07445.
184. Shayne Long­pre, Gre­gory Yauney, Emily Reif et al. (2023). A Pre­trainer's Guide to Train­ing Data: Mea­sur­ing the Effects of Data Age, Domain Cov­er­age, Qual­ity, & Tox­i­c­ity. arXiv:2305.13169.
185. Kather­ine Lee, Daphne Ippolito, Andrew Nys­trom et al. (2021). Dedu­pli­cat­ing Train­ing Data Makes Lan­guage Models Better. arXiv:2107.06499.
186. Nikhil Kand­pal, Eric Wal­lace and Colin Raffel (2022). Dedu­pli­cat­ing Train­ing Data Mit­i­gates Pri­vacy Risks in Lan­guage Models. arXiv:2202.06539.
187. A.Z. Broder (1997). On the resem­blance and con­tain­ment of doc­u­ments. Pro­ceed­ings. Com­pres­sion and Com­plex­ity of SEQUENCES 1997 (Cat. No.97TB100171).
188. Jure Leskovec, Anand Rajara­man and Jef­frey D. Ullman (2020). Mining of Mas­sive Datasets. Cam­bridge Uni­ver­sity Press (3rd ed., free online).
189. Leo Gao, Stella Bider­man, Sid Black et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Lan­guage Mod­el­ing. arXiv:2101.00027.
190. Mau­rice Weber, Daniel Fu, Quentin Anthony et al. (2024). RedPajama: an Open Dataset for Train­ing Large Lan­guage Models. arXiv:2411.12372.
191. Essen­tial AI, Andrew Hojel, Michael Pust et al. (2025). Essen­tial-Web v1.0: 24T tokens of orga­nized web data. arXiv:2506.14111.
192. Nikhil Kand­pal, Brian Lester, Colin Raffel et al. (2025). The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text. arXiv:2506.05209.
193. Ren Ma, Jiantao Qiu, Chao Xu et al. (2025). AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser. arXiv:2511.16397.
194. Jenna Rus­sell, Ben Glick­en­haus, Kather­ine Thai et al. (2026). How Much Is an AI Token Worth? Scal­ing Laws for Wild AI-Gen­er­ated Web Text. arXiv:2609.40295.
195. Sang Michael Xie, Hieu Pham, Xuanyi Dong et al. (2023). DoReMi: Opti­miz­ing Data Mix­tures Speeds Up Lan­guage Model Pre­train­ing. arXiv:2305.10429.
196. Marah Abdin, Jyoti Aneja, Harki­rat Behl et al. (2024). Phi-4 Tech­ni­cal Report. arXiv:2412.08905.
197. Loubna Ben Allal, Anton Lozhkov, Elie Bak­ouch et al. (2025). SmolLM2: When Smol Goes Big -- Data-Cen­tric Train­ing of a Small Lan­guage Model. arXiv:2502.02737.
198. An Yang, Anfeng Li, Baosong Yang et al. (2025). Qwen3 Tech­ni­cal Report. arXiv:2505.09388.
199. Anton Lozhkov, Ray­mond Li, Loubna Ben Allal et al. (2024). StarCoder 2 and The Stack v2: The Next Gen­er­a­tion. arXiv:2402.19173.
200. Jiasheng Ye, Peiju Liu, Tianx­i­ang Sun et al. (2024). Data Mixing Laws: Opti­miz­ing Data Mix­tures by Pre­dict­ing Lan­guage Mod­el­ing Per­for­mance. arXiv:2403.16952.
201. Qian Liu, Xiaosen Zheng, Niklas Muen­nighoff et al. (2024). RegMix: Data Mix­ture as Regres­sion for Lan­guage Model Pre-train­ing. arXiv:2407.01492.
202. Mayee F. Chen, Tyler Murray, David Heine­man et al. (2026). Olmix: A Frame­work for Data Mixing Through­out LM Devel­op­ment. arXiv:2602.12237.
203. Mayee F. Chen, Michael Y. Hu, Nicholas Lourie, Kyunghyun Cho and Christo­pher Ré (2024). Aioli: A Uni­fied Opti­miza­tion Frame­work for Lan­guage Model Data Mixing. arXiv:2411.05735.
204. Mustafa Shukor, Louis Bethune, Dan Bus­bridge et al. (2025). Scal­ing Laws for Opti­mal Data Mix­tures. arXiv:2507.09404.
205. Viraat Aryabumi, Yixuan Su, Ray­mond Ma et al. (2024). To Code, or Not To Code? Explor­ing Impact of Code in Pre-train­ing. arXiv:2408.10914.
206. Keiran Paster, Marco Dos Santos, Zhangir Azer­bayev and Jimmy Ba (2023). OpenWebMath: An Open Dataset of High-Qual­ity Math­e­mat­i­cal Web Text. arXiv:2310.06786.
207. Zhi­hong Shao, Peiyi Wang, Qihao Zhu et al. (2024). DeepSeekMath: Push­ing the Limits of Math­e­mat­i­cal Rea­son­ing in Open Lan­guage Models. arXiv:2402.03300.
208. Cody Blak­eney, Man­sheej Paul, Brett W. Larsen, Sean Owen and Jonathan Fran­kle (2024). Does your data spark joy? Per­for­mance gains from domain upsam­pling at the end of train­ing. arXiv:2406.03476.
209. Niklas Muen­nighoff, Alexan­der M. Rush, Boaz Barak et al. (2023). Scal­ing Data-Con­strained Lan­guage Models. arXiv:2305.16264.
210. Danny Her­nan­dez, Tom Brown, Tom Con­erly et al. (2022). Scal­ing Laws and Inter­pretabil­ity of Learn­ing from Repeated Data. arXiv:2205.10487.
211. Fuzhao Xue, Yao Fu, Wangchun­shu Zhou, Zang­wei Zheng and Yang You (2023). To Repeat or Not To Repeat: Insights from Scal­ing LLM under Token-Crisis. arXiv:2305.13230.
212. Konwoo Kim, Suhas Kotha, Percy Liang and Tat­sunori Hashimoto (2025). Pre-train­ing under infi­nite com­pute. arXiv:2509.14786.
213. Mihir Prab­hude­sai, Mengn­ing Wu, Amir Zadeh, Kate­rina Fragki­adaki and Deepak Pathak (2025). Dif­fu­sion Beats Autore­gres­sive in Data-Con­strained Set­tings. arXiv:2507.15857.
214. Suriya Gunasekar, Yi Zhang, Jyoti Aneja et al. (2023). Text­books Are All You Need. arXiv:2306.11644.
215. Pratyush Maini, Skyler Seto, He Bai et al. (2024). Rephras­ing the Web: A Recipe for Com­pute and Data-Effi­cient Lan­guage Mod­el­ing. arXiv:2401.16380.
216. Ilia Shu­mailov, Zakhar Shu­maylov, Yiren Zhao et al. (2024). AI models col­lapse when trained on recur­sively gen­er­ated data. Nature.
217. Matthias Ger­st­grasser, Rylan Scha­ef­fer, Apra­tim Dey et al. (2024). Is Model Col­lapse Inevitable? Break­ing the Curse of Recur­sion by Accu­mu­lat­ing Real and Syn­thetic Data. arXiv:2404.01413.
218. Elvis Dohma­tob, Yun­zhen Feng, Arjun Sub­ra­mo­nian and Julia Kempe (2024). Strong Model Col­lapse. arXiv:2410.04840.
219. Shayne Long­pre, Robert Mahari, Ariel Lee et al. (2024). Con­sent in Crisis: The Rapid Decline of the AI Data Com­mons. arXiv:2407.14933.
220. Shayne Long­pre, Robert Mahari, Anthony Chen et al. (2023). The Data Prove­nance Ini­tia­tive: A Large Scale Audit of Dataset Licens­ing & Attri­bu­tion in AI. arXiv:2310.16787.
221. OpenAI (2022). tik­to­ken. GitHub repos­i­tory.
222. Rico Sen­nrich, Barry Haddow and Alexan­dra Birch (2015). Neural Machine Trans­la­tion of Rare Words with Sub­word Units. arXiv:1508.07909.
223. OpenAI (2024). tik­to­ken/tik­to­ken_ext/openai_public.py at main · openai/tik­to­ken. GitHub repos­i­tory (source file).
224. Hug­ging Face (2025). Tok­eniza­tion algo­rithms. Trans­form­ers doc­u­men­ta­tion.
225. Sander Land and Max Bar­tolo (2024). Fish­ing for Magikarp: Auto­mat­i­cally Detect­ing Under-trained Tokens in Large Lan­guage Models. arXiv:2405.05417.
226. Pre­ston Fire­stone, Shub­ham Ugare, Gagan­deep Singh and Sasa Mis­ailovic (2025). UTF-8 Plumb­ing: Byte-level Tok­eniz­ers Unavoid­ably Enable LLMs to Gen­er­ate Ill-formed UTF-8. arXiv:2511.05578.
227. Kaj Bostrom and Greg Dur­rett (2020). Byte Pair Encod­ing is Sub­op­ti­mal for Lan­guage Model Pre­train­ing. arXiv:2004.03720.
228. Taku Kudo and John Richard­son (2018). SentencePiece: A simple and lan­guage inde­pen­dent sub­word tok­enizer and deto­k­enizer for Neural Text Pro­cess­ing. arXiv:1808.06226.
229. Hugo Tou­vron, Louis Martin, Kevin Stone et al. (2023). Llama 2: Open Foun­da­tion and Fine-Tuned Chat Models. arXiv:2307.09288.
230. Gemma Team, Thomas Mes­nard, Cas­sidy Hardin et al. (2024). Gemma: Open Models Based on Gemini Research and Tech­nol­ogy. arXiv:2403.08295.
231. Taku Kudo (2018). Sub­word Reg­u­lar­iza­tion: Improv­ing Neural Net­work Trans­la­tion Models with Mul­ti­ple Sub­word Can­di­dates. arXiv:1804.10959.
232. Mike Schus­ter and Kaisuke Naka­jima (2012). Japa­nese and Korean voice search. 2012 IEEE Inter­na­tional Con­fer­ence on Acous­tics, Speech and Signal Pro­cess­ing (ICASSP).
233. Hug­ging Face (2024). WordPiece tok­eniza­tion. Hug­ging Face LLM Course, Chap­ter 6.
234. Ivan Provilkov, Dmitrii Emelia­nenko and Elena Voita (2019). BPE-Dropout: Simple and Effec­tive Sub­word Reg­u­lar­iza­tion. arXiv:1910.13267.
235. Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh and Noah A. Smith (2024). Data Mix­ture Infer­ence: What do BPE Tok­eniz­ers Reveal about their Train­ing Data?. arXiv:2407.16607.
236. Meta AI (2024). Intro­duc­ing Meta Llama 3: The most capa­ble openly avail­able LLM to date. Blog post.
237. Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez et al. (2024). Byte Latent Trans­former: Patches Scale Better Than Tokens. arXiv:2412.09871.
238. OpenAI (2024). tik­to­ken/tik­to­ken/core.py at main · openai/tik­to­ken. GitHub repos­i­tory (source file).
239. Hug­ging Face (2025). Chat tem­plates. Trans­form­ers doc­u­men­ta­tion.
240. Buu Phan, Bran­don Amos, Itai Gat et al. (2024). Exact Byte-Level Prob­a­bil­i­ties from Tok­enized Lan­guage Models for FIM-Tasks and Model Ensem­bles. arXiv:2410.09303.
241. Alisa Liu, Jonathan Hayase, Valentin Hof­mann et al. (2025). SuperBPE: Space Travel for Lan­guage Models. arXiv:2503.13423.
242. Gau­tier Dagan, Gabriel Syn­naeve and Bap­tiste Rozière (2024). Get­ting the most out of your tok­enizer for pre-train­ing and domain adap­ta­tion. arXiv:2402.01035.
243. Alek­san­dar Petrov, Emanuele La Malfa, Philip H. S. Torr and Adel Bibi (2023). Lan­guage Model Tok­eniz­ers Intro­duce Unfair­ness Between Lan­guages. arXiv:2305.15425.
244. Ore­vaoghene Ahia, Sachin Kumar, Hila Gonen et al. (2023). Do All Lan­guages Cost the Same? Tok­eniza­tion in the Era of Com­mer­cial Lan­guage Models. arXiv:2305.13707.
245. Mehdi Ali, Michael Fromm, Klau­dia Thell­mann et al. (2023). Tok­enizer Choice For LLM Train­ing: Neg­li­gi­ble or Cru­cial?. arXiv:2310.08754.
246. Aaditya K. Singh and DJ Strouse (2024). Tok­eniza­tion counts: the impact of tok­eniza­tion on arith­metic in fron­tier LLMs. arXiv:2402.14903.
247. Jes­sica Rum­be­low and Matthew Watkins (2023). SolidGoldMagikarp (plus, prompt gen­er­a­tion). LessWrong.
248. Woojin Chung and Jeonghoon Kim (2025). Exploit­ing Vocab­u­lary Fre­quency Imbal­ance in Lan­guage Model Pre-train­ing. arXiv:2508.15390.
249. Chao­fan Tao, Qian Liu, Longxu Dou et al. (2024). Scal­ing Laws with Vocab­u­lary: Larger Models Deserve Larger Vocab­u­lar­ies. arXiv:2407.13623.
250. Craig W. Schmidt, Varshini Reddy, Haoran Zhang et al. (2024). Tok­eniza­tion Is More Than Com­pres­sion. arXiv:2402.18376.
251. Hongzhi Huang, Defa Zhu, Banggu Wu et al. (2025). Over-Tok­enized Trans­former: Vocab­u­lary is Gen­er­ally Worth Scal­ing. arXiv:2501.16975.
252. Théo Gigant, Bowen Peng and Jef­frey Ques­nelle (2026). Decou­pling the Ben­e­fits of Sub­word Tok­eniza­tion for Lan­guage Model Train­ing via Byte-level Sim­u­la­tion. arXiv:2604.27263.
253. Lint­ing Xue, Aditya Barua, Noah Con­stant et al. (2021). ByT5: Towards a token-free future with pre-trained byte-to-byte models. arXiv:2105.13626.
254. Lili Yu, Dániel Simig, Colin Fla­herty et al. (2023). MEGABYTE: Pre­dict­ing Mil­lion-byte Sequences with Mul­ti­scale Trans­form­ers. arXiv:2305.07185.
255. Sukjun Hwang, Bran­don Wang and Albert Gu (2025). Dynamic Chunk­ing for End-to-End Hier­ar­chi­cal Sequence Mod­el­ing. arXiv:2507.07955.
256. Junx­iong Wang, Tushaar Gan­gavarapu, Jing Nathan Yan and Alexan­der M. Rush (2024). MambaByte: Token-free Selec­tive State Space Model. arXiv:2401.13660.
257. Math­urin Videau, Badr Youbi Idrissi, Alessan­dro Leite et al. (2025). From Bytes to Ideas: Lan­guage Mod­el­ing with Autore­gres­sive U-Nets. arXiv:2506.14761.
258. Ian Mag­nus­son, Nguyen Tai, Ben Bogin et al. (2025). DataDecide: How to Pre­dict Best Pre­train­ing Data with Small Exper­i­ments. arXiv:2504.11393.
259. David Heine­man, Valentin Hof­mann, Ian Mag­nus­son et al. (2025). Signal and Noise: A Frame­work for Reduc­ing Uncer­tainty in Lan­guage Model Eval­u­a­tion. arXiv:2508.13144.
260. Andrej Karpa­thy (2024). karpa­thy/build-nanogpt: Video+code lec­ture on build­ing nanoGPT from scratch. GitHub repos­i­tory.
261. Alec Rad­ford, Karthik Narasimhan, Tim Sal­i­mans and Ilya Sutskever (2018). Improv­ing Lan­guage Under­stand­ing by Gen­er­a­tive Pre-Train­ing. OpenAI tech­ni­cal report.
262. Thomas Wang, Adam Roberts, Daniel Hess­low et al. (2022). What Lan­guage Model Archi­tec­ture and Pre­train­ing Objec­tive Work Best for Zero-Shot Gen­er­al­iza­tion?. arXiv:2204.05832.
263. Yi Tay, Mostafa Dehghani, Vinh Q. Tran et al. (2022). UL2: Uni­fy­ing Lan­guage Learn­ing Par­a­digms. arXiv:2205.05131.
264. Biao Zhang, Fedor Moi­seev, Joshua Ainslie et al. (2025). Encoder-Decoder Gemma: Improv­ing the Qual­ity-Effi­ciency Trade-Off via Adap­ta­tion. arXiv:2504.06225.
265. Gemma Team, Aish­warya Kamath, Johan Ferret et al. (2025). Gemma 3 Tech­ni­cal Report. arXiv:2503.19786.
266. Ashish Vaswani, Noam Shazeer, Niki Parmar et al. (2017). Atten­tion Is All You Need. arXiv:1706.03762.
267. Yann N. Dauphin, Angela Fan, Michael Auli and David Grang­ier (2016). Lan­guage Mod­el­ing with Gated Con­vo­lu­tional Net­works. arXiv:1612.08083.
268. Gemma Team, Mor­gane Riv­iere, Shreya Pathak et al. (2024). Gemma 2: Improv­ing Open Lan­guage Models at a Prac­ti­cal Size. arXiv:2408.00118.
269. Ofir Press and Lior Wolf (2016). Using the Output Embed­ding to Improve Lan­guage Models. arXiv:1608.05859.
270. Mitchell Worts­man, Peter J. Liu, Lechao Xiao et al. (2023). Small-scale prox­ies for large-scale Trans­former train­ing insta­bil­i­ties. arXiv:2309.14322.
271. Chameleon Team (2024). Chameleon: Mixed-Modal Early-Fusion Foun­da­tion Models. arXiv:2405.09818.
272. Ruibin Xiong, Yun­chang Yang, Di He et al. (2020). On Layer Nor­mal­iza­tion in the Trans­former Archi­tec­ture. arXiv:2002.04745.
273. Wen­fang Sun, Xinyuan Song, Pengx­i­ang Li et al. (2025). The Curse of Depth in Large Lan­guage Models. arXiv:2502.05795.
274. Andrey Gromov, Kushal Tiru­mala, Hassan Shapourian, Paolo Glo­rioso and Daniel A. Roberts (2024). The Unrea­son­able Inef­fec­tive­ness of the Deeper Layers. arXiv:2403.17887.
275. Jimmy Lei Ba, Jamie Ryan Kiros and Geof­frey E. Hinton (2016). Layer Nor­mal­iza­tion. arXiv:1607.06450.
276. Biao Zhang and Rico Sen­nrich (2019). Root Mean Square Layer Nor­mal­iza­tion. arXiv:1910.07467.
277. Shuangfei Zhai, Tatiana Likhoma­nenko, Etai Lit­twin et al. (2023). Sta­bi­liz­ing Trans­former Train­ing by Pre­vent­ing Atten­tion Entropy Col­lapse. arXiv:2303.06296.
278. Mostafa Dehghani, Josip Djo­longa, Basil Mustafa et al. (2023). Scal­ing Vision Trans­form­ers to 22 Bil­lion Param­e­ters. arXiv:2302.05442.
279. Alex Henry, Prud­hvi Raj Dacha­pally, Shub­ham Pawar and Yuxuan Chen (2020). Query-Key Nor­mal­iza­tion for Trans­form­ers. arXiv:2010.04245.
280. Niklas Muen­nighoff, Luca Sol­daini, Dirk Groen­eveld et al. (2024). OLMoE: Open Mix­ture-of-Experts Lan­guage Models. arXiv:2409.02060.
281. Bowen Yang, Bharat Venkitesh, Dwarak Talupuru et al. (2025). Rope to Nope and Back Again: A New Hybrid Atten­tion Strat­egy. arXiv:2501.18795.
282. DeepSeek-AI, Aixin Liu, Bei Feng et al. (2024). DeepSeek-V2: A Strong, Eco­nom­i­cal, and Effi­cient Mix­ture-of-Experts Lan­guage Model. arXiv:2405.04434.
283. Ebte­sam Almazrouei, Hamza Alobei­dli, Abdu­laziz Alshamsi et al. (2023). The Falcon Series of Open Lan­guage Models. arXiv:2311.16867.
284. Oleg Rybakov, Mike Chrzanowski, Peter Dykas, Jinze Xue and Ben Lanir (2024). Meth­ods of improv­ing LLM train­ing sta­bil­ity. arXiv:2410.16682.
285. Jeonghoon Kim, Byeongchan Lee, Cheon­bok Park et al. (2025). Peri-LN: Revis­it­ing Nor­mal­iza­tion Layer in the Trans­former Archi­tec­ture. arXiv:2502.02732.
286. Qwen, An Yang, Baosong Yang et al. (2024). Qwen2.5 Tech­ni­cal Report. arXiv:2412.15115.
287. Zihan Qiu, Zekun Wang, Bo Zheng et al. (2025). Gated Atten­tion for Large Lan­guage Models: Non-lin­ear­ity, Spar­sity, and Atten­tion-Sink-Free. arXiv:2505.06708.
288. Sharan Narang, Hyung Won Chung, Yi Tay et al. (2021). Do Trans­former Mod­i­fi­ca­tions Trans­fer Across Imple­men­ta­tions and Appli­ca­tions?. arXiv:2102.11972.
289. Yi Tay, Mostafa Dehghani, Samira Abnar et al. (2022). Scal­ing Laws vs Model Archi­tec­tures: How does Induc­tive Bias Influ­ence Scal­ing?. arXiv:2207.10551.
290. David R. So, Woj­ciech Mańke, Hanx­iao Liu et al. (2021). Primer: Search­ing for Effi­cient Trans­form­ers for Lan­guage Mod­el­ing. arXiv:2109.08668.
291. Nvidia, Bo Adler, Niket Agar­wal et al. (2024). Nemotron-4 340B Tech­ni­cal Report. arXiv:2406.11704.
292. NVIDIA, Aaron Blake­man, Aarti Basant et al. (2025). Nemotron-H: A Family of Accu­rate and Effi­cient Hybrid Mamba-Trans­former Models. arXiv:2504.03624.
293. Yi Tay, Mostafa Dehghani, Jin­feng Rao et al. (2021). Scale Effi­ciently: Insights from Pre-train­ing and Fine-tuning Trans­form­ers. arXiv:2109.10686.
294. Zechun Liu, Chang­sheng Zhao, For­rest Ian­dola et al. (2024). MobileLLM: Opti­miz­ing Sub-bil­lion Param­e­ter Lan­guage Models for On-Device Use Cases. arXiv:2402.14905.
295. 5 Team, Aohan Zeng, Xin Lv et al. (2025). GLM-4.5: Agen­tic, Rea­son­ing, and Coding (ARC) Foun­da­tion Models. arXiv:2508.06471.
296. Albert Q. Jiang, Alexan­dre Sablay­rolles, Arthur Mensch et al. (2023). Mis­tral 7B. arXiv:2310.06825.
297. Guangx­uan Xiao, Yuan­dong Tian, Beidi Chen, Song Han and Mike Lewis (2023). Effi­cient Stream­ing Lan­guage Models with Atten­tion Sinks. arXiv:2309.17453.
298. Ofir Press, Noah A. Smith and Mike Lewis (2021). Train Short, Test Long: Atten­tion with Linear Biases Enables Input Length Extrap­o­la­tion. arXiv:2108.12409.
299. Jian­lin Su, Yu Lu, Shengfeng Pan et al. (2021). RoFormer: Enhanced Trans­former with Rotary Posi­tion Embed­ding. arXiv:2104.09864.
300. Fed­erico Bar­bero, Alex Vitvit­skyi, Chris­tos Perivolaropou­los, Razvan Pas­canu and Petar Veličković (2024). Round and Round We Go! What makes Rotary Posi­tional Encod­ings useful?. arXiv:2410.06205.
301. MiniMax, Aonian Li, Bang­wei Gong et al. (2025). MiniMax-01: Scal­ing Foun­da­tion Models with Light­ning Atten­tion. arXiv:2501.08313.
302. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak and Omer Levy (2022). Trans­former Lan­guage Models with­out Posi­tional Encod­ings Still Learn Posi­tional Infor­ma­tion. arXiv:2203.16634.
303. Meta AI (2025). The Llama 4 herd: The begin­ning of a new era of natively mul­ti­modal AI inno­va­tion. Meta AI blog.
304. Opher Lieber, Barak Lenz, Hofit Bata et al. (2024). Jamba: A Hybrid Trans­former-Mamba Lan­guage Model. arXiv:2403.19887.
305. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan and Geof­frey E. Hinton (1991). Adap­tive Mix­tures of Local Experts. Neural Com­pu­ta­tion.
306. Michael I. Jordan and Robert A. Jacobs (1994). Hier­ar­chi­cal Mix­tures of Experts and the EM Algo­rithm. Neural Com­pu­ta­tion.
307. Noam Shazeer, Azalia Mirho­seini, Krzysztof Maziarz et al. (2017). Out­ra­geously Large Neural Net­works: The Sparsely-Gated Mix­ture-of-Experts Layer. arXiv:1701.06538.
308. Dmitry Lep­ikhin, HyoukJoong Lee, Yuanzhong Xu et al. (2020). GShard: Scal­ing Giant Models with Con­di­tional Com­pu­ta­tion and Auto­matic Shard­ing. arXiv:2006.16668.
309. Nan Du, Yan­ping Huang, Andrew M. Dai et al. (2021). GLaM: Effi­cient Scal­ing of Lan­guage Models with Mix­ture-of-Experts. arXiv:2112.06905.
310. Albert Q. Jiang, Alexan­dre Sablay­rolles, Antoine Roux et al. (2024). Mix­tral of Experts. arXiv:2401.04088.
311. Yanqi Zhou, Tao Lei, Hanx­iao Liu et al. (2022). Mix­ture-of-Experts with Expert Choice Rout­ing. arXiv:2202.09368.
312. Samira Abnar, Har­shay Shah, Dan Bus­bridge et al. (2025). Param­e­ters vs FLOPs: Scal­ing Laws for Opti­mal Spar­sity for Mix­ture-of-Experts Lan­guage Models. arXiv:2501.12370.
313. Lean Wang, Huazuo Gao, Cheng­gang Zhao, Xu Sun and Damai Dai (2024). Aux­il­iary-Loss-Free Load Bal­anc­ing Strat­egy for Mix­ture-of-Experts. arXiv:2408.15664.
314. Barret Zoph, Irwan Bello, Sameer Kumar et al. (2022). ST-MoE: Design­ing Stable and Trans­fer­able Sparse Expert Models. arXiv:2202.08906.
315. Trevor Gale, Deepak Narayanan, Cliff Young and Matei Zaharia (2022). MegaBlocks: Effi­cient Sparse Train­ing with Mix­ture-of-Experts. arXiv:2211.15841.
316. Zihan Qiu, Zeyu Huang, Bo Zheng et al. (2025). Demons in the Detail: On Imple­ment­ing Load Bal­anc­ing Loss for Train­ing Spe­cial­ized Mix­ture-of-Expert Models. arXiv:2501.11873.
317. Damai Dai, Chengqi Deng, Cheng­gang Zhao et al. (2024). DeepSeekMoE: Towards Ulti­mate Expert Spe­cial­iza­tion in Mix­ture-of-Experts Lan­guage Models. arXiv:2401.06066.
318. Jakub Kra­jew­ski, Jan Ludziejew­ski, Kamil Adam­czewski et al. (2024). Scal­ing Laws for Fine-Grained Mix­ture of Experts. arXiv:2402.07871.
319. Aran Komat­suzaki, Joan Puigcerver, James Lee-Thorp et al. (2022). Sparse Upcy­cling: Train­ing Mix­ture-of-Experts from Dense Check­points. arXiv:2212.05055.
320. Tian­wen Wei, Bo Zhu, Liang Zhao et al. (2024). Sky­work-MoE: A Deep Dive into Train­ing Tech­niques for Mix­ture-of-Experts Lan­guage Models. arXiv:2406.06563.
321. Ange­los Katharopou­los, Apoorv Vyas, Niko­laos Pappas and François Fleuret (2020). Trans­form­ers are RNNs: Fast Autore­gres­sive Trans­form­ers with Linear Atten­tion. arXiv:2006.16236.
322. Imanol Schlag, Kazuki Irie and Jürgen Schmid­hu­ber (2021). Linear Trans­form­ers Are Secretly Fast Weight Pro­gram­mers. arXiv:2102.11174.
323. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen and Yoon Kim (2024). Par­al­leliz­ing Linear Trans­form­ers with the Delta Rule over Sequence Length. arXiv:2406.06484.
324. Yutao Sun, Li Dong, Shao­han Huang et al. (2023). Reten­tive Net­work: A Suc­ces­sor to Trans­former for Large Lan­guage Models. arXiv:2307.08621.
325. Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim (2023). Gated Linear Atten­tion Trans­form­ers with Hard­ware-Effi­cient Train­ing. arXiv:2312.06635.
326. Tri Dao and Albert Gu (2024). Trans­form­ers are SSMs: Gen­er­al­ized Models and Effi­cient Algo­rithms Through Struc­tured State Space Dual­ity. arXiv:2405.21060.
327. Albert Gu and Tri Dao (2023). Mamba: Linear-Time Sequence Mod­el­ing with Selec­tive State Spaces. arXiv:2312.00752.
328. Songlin Yang, Jan Kautz and Ali Hatamizadeh (2024). Gated Delta Net­works: Improv­ing Mamba2 with Delta Rule. arXiv:2412.06464.
329. Simran Arora, Sabri Eyuboglu, Aman Timalsina et al. (2023). Zool­ogy: Mea­sur­ing and Improv­ing Recall in Effi­cient Lan­guage Models. arXiv:2312.04927.
330. Samy Jelassi, David Brand­fon­brener, Sham M. Kakade and Eran Malach (2024). Repeat After Me: Trans­form­ers are Better than State Space Models at Copy­ing. arXiv:2402.01032.
331. Roger Wal­effe, Wonmin Byeon, Duncan Riach et al. (2024). An Empir­i­cal Study of Mamba-based Lan­guage Models. arXiv:2406.07887.
332. Dustin Wang, Rui-Jie Zhu, Steven Abreu et al. (2025). A Sys­tem­atic Anal­y­sis of Hybrid Linear Atten­tion. arXiv:2507.06457.
333. Kimi Team, Yu Zhang, Zongyu Lin et al. (2025). Kimi Linear: An Expres­sive, Effi­cient Atten­tion Archi­tec­ture. arXiv:2510.26692.
334. Lil­iang Ren, Yang Liu, Yadong Lu et al. (2024). Samba: Simple Hybrid State Space Models for Effi­cient Unlim­ited Con­text Lan­guage Mod­el­ing. arXiv:2406.07522.
335. Fabian Gloeckle, Badr Youbi Idrissi, Bap­tiste Rozière, David Lopez-Paz and Gabriel Syn­naeve (2024). Better & Faster Large Lan­guage Models via Multi-token Pre­dic­tion. arXiv:2404.19737.
336. Shen Nie, Fengqi Zhu, Zebin You et al. (2025). Large Lan­guage Dif­fu­sion Models. arXiv:2502.09992.
337. Diederik P. Kingma and Jimmy Ba (2014). Adam: A Method for Sto­chas­tic Opti­miza­tion. arXiv:1412.6980.
338. Ilya Loshchilov and Frank Hutter (2017). Decou­pled Weight Decay Reg­u­lar­iza­tion. arXiv:1711.05101.
339. Shane Bergsma, Nolan Dey, Gur­preet Gosal et al. (2025). Straight to Zero: Why Lin­early Decay­ing the Learn­ing Rate to Zero Works Best for LLMs. arXiv:2502.15938.
340. Xi Wang and Lau­rence Aitchi­son (2024). How to set AdamW's weight decay as you scale model and dataset size. arXiv:2405.13698.
341. Shane Bergsma, Nolan Dey, Gur­preet Gosal et al. (2025). Power Lines: Scal­ing Laws for Weight Decay and Batch Size in LLM Pre-train­ing. arXiv:2505.13738.
342. Igor Moly­bog, Peter Albert, Moya Chen et al. (2023). A Theory on Adam Insta­bil­ity in Large-Scale Machine Learn­ing. arXiv:2304.09871.
343. Martin Marek, Sanae Lotfi, Aditya Soma­sun­daram, Andrew Gordon Wilson and Micah Gold­blum (2025). Small Batch Size Train­ing for Lan­guage Models: When Vanilla SGD Works, and Why Gra­di­ent Accu­mu­la­tion Is Waste­ful. arXiv:2507.07101.
344. Razvan Pas­canu, Tomas Mikolov and Yoshua Bengio (2012). On the dif­fi­culty of train­ing Recur­rent Neural Net­works. arXiv:1211.5063.
345. Susan Zhang, Stephen Roller, Naman Goyal et al. (2022). OPT: Open Pre-trained Trans­former Lan­guage Models. arXiv:2205.01068.
346. Aaron Defazio (2025). Why Gra­di­ents Rapidly Increase Near the End of Train­ing. arXiv:2506.02285.
347. Rosie Zhao, Depen Mor­wani, David Brand­fon­brener, Nikhil Vyas and Sham Kakade (2024). Decon­struct­ing What Makes a Good Opti­mizer for Lan­guage Models. arXiv:2407.07972.
348. Liyuan Liu, Haom­ing Jiang, Pengcheng He et al. (2019). On the Vari­ance of the Adap­tive Learn­ing Rate and Beyond. arXiv:1908.03265.
349. Jerry Ma and Denis Yarats (2019). On the ade­quacy of untuned warmup for adap­tive opti­miza­tion. arXiv:1910.04209.
350. Dayal Singh Kalra and Mais­sam Barkeshli (2024). Why Warmup the Learn­ing Rate? Under­ly­ing Mech­a­nisms and Improve­ments. arXiv:2406.09405.
351. Justin Gilmer, Behrooz Ghor­bani, Ankush Garg et al. (2021). A Loss Cur­va­ture Per­spec­tive on Train­ing Insta­bil­ity in Deep Learn­ing. arXiv:2110.04369.
352. Ilya Loshchilov and Frank Hutter (2016). SGDR: Sto­chas­tic Gra­di­ent Descent with Warm Restarts. arXiv:1608.03983.
353. Alexan­der Hägele, Elie Bak­ouch, Atli Kosson et al. (2024). Scal­ing Laws and Com­pute-Opti­mal Train­ing Beyond Fixed Train­ing Dura­tions. arXiv:2405.18392.
354. Shengding Hu, Yuge Tu, Xu Han et al. (2024). MiniCPM: Unveil­ing the Poten­tial of Small Lan­guage Models with Scal­able Train­ing Strate­gies. arXiv:2404.06395.
355. Aaron Defazio, Xingyu Alice Yang, Harsh Mehta et al. (2024). The Road Less Sched­uled. arXiv:2405.15682.
356. Priya Kasim­beg, Frank Schnei­der, Runa Eschen­hagen et al. (2025). Accel­er­at­ing Neural Net­work Train­ing: An Anal­y­sis of the AlgoPerf Com­pe­ti­tion. arXiv:2502.15015.
357. Kaiyue Wen, Zhiyuan Li, Jason Wang et al. (2024). Under­stand­ing Warmup-Stable-Decay Learn­ing Rates: A River Valley Loss Land­scape Per­spec­tive. arXiv:2410.05192.
358. Adam Ibrahim, Ben­jamin Thérien, Kshi­tij Gupta et al. (2024). Simple and Scal­able Strate­gies to Con­tin­u­ally Pre-train Large Lan­guage Models. arXiv:2403.08763.
359. Howe Tissue, Venus Wang and Lu Wang (2024). Scal­ing Law with Learn­ing Rate Anneal­ing. arXiv:2408.11029.
360. Sam McCandlish, Jared Kaplan, Dario Amodei and OpenAI Dota Team (2018). An Empir­i­cal Model of Large-Batch Train­ing. arXiv:1812.06162.
361. Hanlin Zhang, Depen Mor­wani, Nikhil Vyas et al. (2024). How Does Crit­i­cal Batch Size Scale in Pre-train­ing?. arXiv:2410.21676.
362. Houyi Li, Wen­zhen Zheng, Qiufeng Wang et al. (2025). Pre­dictable Scale: Part I, Step Law -- Opti­mal Hyper­pa­ram­e­ter Scal­ing Law in Large Lan­guage Model Pre­train­ing. arXiv:2503.04715.
363. William Mer­rill, Shane Arora, Dirk Groen­eveld and Han­naneh Hajishirzi (2025). Crit­i­cal Batch Size Revis­ited: A Simple Empir­i­cal Approach to Large-Batch Lan­guage Model Train­ing. arXiv:2505.23971.
364. Alexan­dru Meterez, Depen Mor­wani, Jingfeng Wu et al. (2025). Seesaw: Accel­er­at­ing Train­ing by Bal­anc­ing Learn­ing Rate and Batch Size Sched­ul­ing. arXiv:2510.14717.
365. Samuel L. Smith, Pieter-Jan Kin­der­mans, Chris Ying and Quoc V. Le (2017). Don't Decay the Learn­ing Rate, Increase the Batch Size. arXiv:1711.00489.
366. Priya Goyal, Piotr Dollár, Ross Gir­shick et al. (2017). Accu­rate, Large Mini­batch SGD: Train­ing ImageNet in 1 Hour. arXiv:1706.02677.
367. Sad­hika Mal­ladi, Kaifeng Lyu, Abhishek Pan­i­grahi and San­jeev Arora (2022). On the SDEs and Scal­ing Rules for Adap­tive Gra­di­ent Algo­rithms. arXiv:2205.10287.
368. Shuaipeng Li, Peng­hao Zhao, Hailin Zhang et al. (2024). Surge Phe­nom­e­non in Opti­mal Learn­ing Rate and Batch Size Scal­ing. arXiv:2405.14578.
369. Bruno Mlodoze­niec, Pierre Ablin, Louis Béthune et al. (2025). Com­pleted Hyper­pa­ram­e­ter Trans­fer across Mod­ules, Width, Depth, Batch and Dura­tion. arXiv:2512.22382.
370. Xingyu Dang, Kaiyue Wen and Sad­hika Mal­ladi (2026). The Best Opti­mizer Depends on Batch Size. arXiv:2610.08975.
371. Christo­pher J. Shal­lue, Jae­hoon Lee, Joseph Antognini et al. (2018). Mea­sur­ing the Effects of Data Par­al­lel­ism on Neural Net­work Train­ing. arXiv:1811.03600.
372. Greg Yang, Edward J. Hu, Igor Babuschkin et al. (2022). Tensor Pro­grams V: Tuning Large Neural Net­works via Zero-Shot Hyper­pa­ram­e­ter Trans­fer. arXiv:2203.03466.
373. Greg Yang and Edward J. Hu (2020). Fea­ture Learn­ing in Infi­nite-Width Neural Net­works. arXiv:2011.14522.
374. Arthur Jacot, Franck Gabriel and Clément Hon­gler (2018). Neural Tan­gent Kernel: Con­ver­gence and Gen­er­al­iza­tion in Neural Net­works. arXiv:1806.07572.
375. Lenaic Chizat, Edouard Oyal­lon and Fran­cis Bach (2018). On Lazy Train­ing in Dif­fer­en­tiable Pro­gram­ming. arXiv:1812.07956.
376. Greg Yang, James B. Simon and Jeremy Bern­stein (2023). A Spec­tral Con­di­tion for Fea­ture Learn­ing. arXiv:2310.17813.
377. Nolan Dey, Gur­preet Gosal, Zhim­ing et al. (2023). Cere­bras-GPT: Open Com­pute-Opti­mal Lan­guage Models Trained on the Cere­bras Wafer-Scale Clus­ter. arXiv:2304.03208.
378. Nolan Dey, Quentin Anthony and Joel Hes­t­ness (2024). The Prac­ti­tioner's Guide to the Max­i­mal Update Param­e­ter­i­za­tion. Cere­bras blog.
379. Micro­soft (2022). mup: max­i­mal update parametriza­tion (muP). GitHub repos­i­tory.
380. Nolan Dey, Daria Sobol­eva, Faisal Al-Kha­teeb et al. (2023). BTLM-3B-8K: 7B Param­e­ter Per­for­mance in a 3B Param­e­ter Model. arXiv:2309.11568.
381. Lucas Lingle (2024). An Empir­i­cal Study of μP Learn­ing Rate Trans­fer. arXiv:2404.05728.
382. Char­lie Blake, Con­stantin Eichen­berg, Josef Dean et al. (2024). u-μP: The Unit-Scaled Max­i­mal Update Parametriza­tion. arXiv:2407.17465.
383. Jing­wei Zuo, Maksim Velikanov, Ilyas Chahed et al. (2025). Falcon-H1: A Family of Hybrid-Head Lan­guage Models Redefin­ing Effi­ciency and Per­for­mance. arXiv:2507.22448.
384. Team Cohere, Aakanksha, Arash Ahma­dian et al. (2025). Com­mand A: An Enter­prise-Ready Large Lan­guage Model. arXiv:2504.00698.
385. Atli Kosson, Jeremy Wel­born, Yang Liu, Martin Jaggi and Xi Chen (2025). Weight Decay may matter more than muP for Learn­ing Rate Trans­fer in Prac­tice. arXiv:2510.19093.
386. Katie Everett, Lechao Xiao, Mitchell Worts­man et al. (2024). Scal­ing Expo­nents Across Param­e­ter­i­za­tions and Opti­miz­ers. arXiv:2407.05872.
387. Dayal Singh Kalra and Mais­sam Barkeshli (2026). Quan­ti­fy­ing Hyper­pa­ram­e­ter Trans­fer and the Impor­tance of Embed­ding Layer Learn­ing Rate. arXiv:2605.21486.
388. Greg Yang, Dingli Yu, Chen Zhu and Soufi­ane Hayou (2023). Tensor Pro­grams VI: Fea­ture Learn­ing in Infi­nite-Depth Neural Net­works. arXiv:2310.02244.
389. Blake Bor­de­lon, Lorenzo Noci, Mufan Bill Li, Boris Hanin and Cengiz Pehle­van (2023). Depth­wise Hyper­pa­ram­e­ter Trans­fer in Resid­ual Net­works: Dynam­ics and Scal­ing Limit. arXiv:2309.16620.
390. Nolan Dey, Bin Claire Zhang, Lorenzo Noci et al. (2025). Don't be lazy: CompleteP enables com­pute-effi­cient deep trans­form­ers. arXiv:2505.01618.
391. Kaim­ing He, Xiangyu Zhang, Shao­qing Ren and Jian Sun (2015). Delv­ing Deep into Rec­ti­fiers: Sur­pass­ing Human-Level Per­for­mance on ImageNet Clas­si­fi­ca­tion. arXiv:1502.01852.
392. Moham­mad Shoeybi, Mostofa Pat­wary, Raul Puri et al. (2019). Mega­tron-LM: Train­ing Multi-Bil­lion Param­e­ter Lan­guage Models Using Model Par­al­lel­ism. arXiv:1909.08053.
393. Varun God­bole, George E. Dahl, Justin Gilmer, Christo­pher J. Shal­lue and Zachary Nado (2023). Deep Learn­ing Tuning Play­book. GitHub repos­i­tory.
394. Johan Bjorck, Alon Ben­haim, Vishrav Chaud­hary, Furu Wei and Xia Song (2024). Scal­ing Opti­mal LR Across Token Hori­zons. arXiv:2409.19913.
395. Aohan Zeng, Xiao Liu, Zhengx­iao Du et al. (2022). GLM-130B: An Open Bilin­gual Pre-trained Model. arXiv:2210.02414.
396. Mitchell Worts­man, Tim Dettmers, Luke Zettle­moyer et al. (2023). Stable and low-pre­ci­sion train­ing for large-scale vision-lan­guage models. arXiv:2304.13013.
397. Sho Takase, Shun Kiyono, Sosuke Kobayashi and Jun Suzuki (2023). Spike No More: Sta­bi­liz­ing the Pre-train­ing of Large Lan­guage Models. arXiv:2312.16903.
398. Zaiqian Chen, Aditya Soma­sun­daram, Sophie Li et al. (2026). The Role of Quan­ti­za­tion in Loss Spikes. arXiv:2610.38277.
399. Teven Le Scao, Thomas Wang, Daniel Hess­low et al. (2022). What Lan­guage Model to Train if You Have One Mil­lion GPU Hours?. arXiv:2210.15424.
400. Yun­shui Li, Yiyuan Ma, Shen Yan et al. (2025). Model Merg­ing in Pre-train­ing of Large Lan­guage Models. arXiv:2505.12082.
401. BigScience Work­shop, Teven Le Scao, Angela Fan et al. (2022). BLOOM: A 176B-Param­e­ter Open-Access Mul­ti­lin­gual Lan­guage Model. arXiv:2211.05100.
402. Jiayi Yuan, Hao Li, Xin­heng Ding et al. (2025). Under­stand­ing and Mit­i­gat­ing Numer­i­cal Sources of Non­de­ter­min­ism in LLM Infer­ence. arXiv:2506.09501.
403. Bita Darvish Rouhani, Ritchie Zhao, Ankit More et al. (2023). Microscal­ing Data For­mats for Deep Learn­ing. arXiv:2310.10537.
404. Mingjie Sun, Xinlei Chen, J. Zico Kolter and Zhuang Liu (2024). Mas­sive Acti­va­tions in Large Lan­guage Models. arXiv:2402.17762.
405. Kazuki Fujii, Taishi Naka­mura and Rio Yokota (2024). Bal­anc­ing Speed and Sta­bil­ity: The Trade-offs of FP8 vs. BF16 Train­ing in LLMs. arXiv:2411.08719.
406. Ale­jan­dro Hernández-Cano, Dhia Gar­baya, Imanol Schlag and Martin Jaggi (2025). Towards Fully FP8 GEMM LLM Train­ing at Scale. arXiv:2505.20524.
407. Musa Cim, Sarthak Arora, Poova­iah Palan­gappa et al. (2026). Pre­train­ing large lan­guage models with MXFP4 on Native FP4 Hard­ware. arXiv:2605.09825.
408. Jeremy Bern­stein and Laker New­house (2024). Old Opti­mizer, New Norm: An Anthol­ogy. arXiv:2409.20325.
409. Jingyuan Liu, Jian­lin Su, Xingcheng Yao et al. (2025). Muon is Scal­able for LLM Train­ing. arXiv:2502.16982.
410. Vineet Gupta, Tomer Koren and Yoram Singer (2018). Sham­poo: Pre­con­di­tioned Sto­chas­tic Tensor Opti­miza­tion. arXiv:1802.09568.
411. Hao-Jun Michael Shi, Tsung-Hsien Lee, Shin­taro Iwasaki et al. (2023). A Dis­trib­uted Data-Par­al­lel PyTorch Imple­men­ta­tion of the Dis­trib­uted Sham­poo Opti­mizer for Train­ing Neural Net­works At-Scale. arXiv:2309.06497.
412. Nikhil Vyas, Depen Mor­wani, Rosie Zhao et al. (2024). SOAP: Improv­ing and Sta­bi­liz­ing Sham­poo using Adam. arXiv:2409.11321.
413. Keller Jordan, Yuchen Jin, Vlado Boza et al. (2024). Muon: An opti­mizer for hidden layers in neural net­works. Blog post.
414. Mushir Akhtar, Hazem N. Nounou, Majdi Man­souri and Mohamed N. Nounou (2026). When Does Muon Actu­ally Win? A Paired Data-, Time-, and Tuning-Budget Bench­mark. arXiv:2610.21555.
415. Keller Jordan, Jeremy Bern­stein, Bren­dan Rap­pazzo et al. (2024). modded-nanogpt: Speedrun­ning the NanoGPT base­line. GitHub repos­i­tory README.
416. Hong Liu, Zhiyuan Li, David Hall, Percy Liang and Tengyu Ma (2023). Sophia: A Scal­able Sto­chas­tic Second-order Opti­mizer for Lan­guage Model Pre-train­ing. arXiv:2305.14342.
417. Jean Kad­dour, Oscar Key, Piotr Nawrot, Pasquale Min­ervini and Matt J. Kusner (2023). No Train No Gain: Revis­it­ing Effi­cient Train­ing Algo­rithms For Trans­former-based Lan­guage Models. arXiv:2307.06440.
418. Kaiyue Wen, David Hall, Tengyu Ma and Percy Liang (2025). Fan­tas­tic Pre­train­ing Opti­miz­ers and Where to Find Them. arXiv:2509.02046.
419. George E. Dahl, Frank Schnei­der, Zachary Nado et al. (2023). Bench­mark­ing Neural Net­work Train­ing Algo­rithms. arXiv:2306.07179.
420. Sunny Sanyal, Atula Neerkaje, Jean Kad­dour, Abhishek Kumar and Sujay Sang­havi (2023). Early Weight Aver­ag­ing meets High Learn­ing Rates for LLM Pre-train­ing. arXiv:2306.03241.
421. Changxin Tian, Jia­peng Wang, Qian Zhao et al. (2025). WSM: Decay-Free Learn­ing Rate Sched­ule via Check­point Merg­ing for LLM Pre-train­ing. arXiv:2507.17634.
422. Jeremy M. Cohen, Behrooz Ghor­bani, Shankar Krish­nan et al. (2022). Adap­tive Gra­di­ent Meth­ods at the Edge of Sta­bil­ity. arXiv:2207.14484.
423. Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter and Ameet Tal­walkar (2021). Gra­di­ent Descent on Neural Net­works Typ­i­cally Occurs at the Edge of Sta­bil­ity. arXiv:2103.00065.
424. Cather­ine Olsson, Nelson Elhage, Neel Nanda et al. (2022). In-con­text Learn­ing and Induc­tion Heads. arXiv:2209.11895.
425. Pulkit Gopalani and Wei Hu (2025). What Hap­pens During the Loss Plateau? Under­stand­ing Abrupt Learn­ing in Trans­form­ers. arXiv:2506.13688.
426. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra (2022). Grokking: Gen­er­al­iza­tion Beyond Over­fit­ting on Small Algo­rith­mic Datasets. arXiv:2201.02177.
427. Pree­tum Nakki­ran, Gal Kaplun, Yamini Bansal et al. (2019). Deep Double Descent: Where Bigger Models and More Data Hurt. arXiv:1912.02292.
428. Tom Henighan, Jared Kaplan, Mor Katz et al. (2020). Scal­ing Laws for Autore­gres­sive Gen­er­a­tive Mod­el­ing. arXiv:2010.14701.
429. Joel Hes­t­ness, Sharan Narang, Newsha Ardalani et al. (2017). Deep Learn­ing Scal­ing is Pre­dictable, Empir­i­cally. arXiv:1712.00409.
430. Jonathan S. Rosen­feld, Amir Rosen­feld, Yonatan Belinkov and Nir Shavit (2019). A Con­struc­tive Pre­dic­tion of the Gen­er­al­iza­tion Error Across Scales. arXiv:1909.12673.
431. Utkarsh Sharma and Jared Kaplan (2020). A Neural Scal­ing Law from the Dimen­sion of the Data Man­i­fold. arXiv:2004.10802.
432. Eric J. Michaud, Ziming Liu, Uzay Girit and Max Tegmark (2023). The Quan­ti­za­tion Model of Neural Scal­ing. arXiv:2303.13506.
433. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jae­hoon Lee and Utkarsh Sharma (2021). Explain­ing Neural Scal­ing Laws. arXiv:2102.06701.
434. Marcus Hutter (2021). Learn­ing Curve Theory. arXiv:2102.04074.
435. Alexan­der Mal­oney, Daniel A. Roberts and James Sully (2022). A Solv­able Model of Neural Scal­ing Laws. arXiv:2210.16859.
436. Blake Bor­de­lon, Alexan­der Atanasov and Cengiz Pehle­van (2024). A Dynam­i­cal Model of Neural Scal­ing Laws. arXiv:2402.01092.
437. Elliot Paque­tte, Court­ney Paque­tte, Lechao Xiao and Jef­frey Pen­ning­ton (2024). 4+3 Phases of Com­pute-Opti­mal Neural Scal­ing Laws. arXiv:2405.15074.
438. Francesco Cagnetta, Allan Raventós, Surya Gan­guli and Matthieu Wyart (2026). Deriv­ing Neural Scal­ing Laws from the sta­tis­tics of nat­u­ral lan­guage. arXiv:2602.07488.
439. Marc Finzi, Sanyam Kapoor, Diego Granziol et al. (2025). Com­pute-Opti­mal LLMs Prov­ably Gen­er­al­ize Better With Scale. arXiv:2504.15208.
440. Yizhou Liu and Jeff Gore (2026). Neural Scal­ing Uni­ver­sal­ity: If Expo­nents Are Fixed, Time to Under­stand Coef­fi­cients. arXiv:2606.25008.
441. Xing Han, Liu Ziyin, Suchi Saria and Paul Pu Liang (2026). On the Invari­ance and Gen­er­al­ity of Neural Scal­ing Laws. arXiv:2605.07546.
442. Ethan Caballero, Kshi­tij Gupta, Irina Rish and David Krueger (2022). Broken Neural Scal­ing Laws. arXiv:2210.14891.
443. Ethan Caballero, Priyank Jaini, David Krueger and Irina Rish (2026). Uni­fied Neural Scal­ing Laws. arXiv:2605.26248.
444. Ibrahim Alab­dul­mohsin, Behnam Neyshabur and Xiao­hua Zhai (2022). Revis­it­ing Neural Scal­ing Laws in Lan­guage and Vision. arXiv:2209.06640.
445. Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Gan­guli and Ari S. Morcos (2022). Beyond neural scal­ing laws: beat­ing power law scal­ing via data prun­ing. arXiv:2206.14486.
446. Tamay Besiroglu, Ege Erdil, Matthew Bar­nett and Josh You (2024). Chin­chilla Scal­ing: A repli­ca­tion attempt. arXiv:2404.10102.
447. Rylan Scha­ef­fer, Noam Levi, Andreas Kirsch et al. (2025). Eval­u­at­ing the Robust­ness of Chin­chilla Com­pute-Opti­mal Scal­ing. arXiv:2509.23963.
448. Joshua Shay Kricheli, Alexan­der Lawrence Reid, Souma­jy­oti Sarkar, Venkata Gandikota and Paulo Shakar­ian (2026). Tokens-per-Param­e­ter Cov­er­age Is Crit­i­cal for Robust LLM Scal­ing Law Extrap­o­la­tion. arXiv:2605.08541.
449. Sean McLeish, John Kirchen­bauer, David Yu Miller et al. (2025). Gem­stones: A Model Suite for Multi-Faceted Scal­ing Laws. arXiv:2502.06857.
450. Leshem Choshen, Yang Zhang and Jacob Andreas (2024). A Hitch­hiker's Guide to Scal­ing Law Esti­ma­tion. arXiv:2410.11840.
451. Mar­garet Li, Sneha Kudugunta and Luke Zettle­moyer (2025). (Mis)Fit­ting: A Survey of Scal­ing Laws. arXiv:2502.18969.
452. Bruno Mlodoze­niec and Spencer Frei (2026). Ambigu­ous Fits: The Rashomon Effect and Chal­lenges in Scal­ing Law Esti­ma­tion. arXiv:2610.34514.
453. Vansh Ramani and Shourya Vir Jain (2026). On the Opti­mizer Depen­dence of Neural Scal­ing Laws. arXiv:2605.29387.
454. Sijie Li, Shanda Li, Haowei Lin et al. (2026). Spend Less, Fit Better: Budget-Effi­cient Scal­ing Law Fit­ting via Active Exper­i­ment Selec­tion. arXiv:2604.22753.
455. Houyi Li, Wen­zhen Zheng, Qiufeng Wang et al. (2025). Pre­dictable Scale: Part II, Farseer: A Refined Scal­ing Law in Large Lan­guage Models. arXiv:2506.10972.
456. Samir Yitzhak Gadre, Geor­gios Smyr­nis, Vaishaal Shankar et al. (2024). Lan­guage models scale reli­ably with over-train­ing and on down­stream tasks. arXiv:2403.08540.
457. Pablo Vil­lalo­bos, Anson Ho, Jaime Sevilla et al. (2022). Will we run out of data? Limits of LLM scal­ing based on human-gen­er­ated data. arXiv:2211.04325.
458. Tan­ishq Kumar, Zachary Ankner, Ben­jamin F. Spec­tor et al. (2024). Scal­ing Laws for Pre­ci­sion. arXiv:2411.04330.
459. Song Bian, Ming­hao Yan and Shiv­aram Venkatara­man (2025). Scal­ing Infer­ence-Effi­cient Lan­guage Models. arXiv:2501.18107.
460. Justin Lovelace, Chris­tian Belardi, Sri­vatsa Kun­durthy, Shriya Sud­hakar and Kilian Q. Wein­berger (2026). Pre­scrip­tive Scal­ing Laws for Data Con­strained Train­ing. arXiv:2605.01640.
461. Aidan Clark, Diego de las Casas, Aure­lia Guy et al. (2022). Uni­fied Scal­ing Laws for Routed Lan­guage Models. arXiv:2202.01169.
462. Jan Ludziejew­ski, Maciej Pióro, Jakub Kra­jew­ski et al. (2025). Joint MoE Scal­ing Laws: Mix­ture of Experts Can Be Memory Effi­cient. arXiv:2502.05172.
463. Guo­liang Zhao, Yuhan Fu, Shuaipeng Li et al. (2025). Towards a Com­pre­hen­sive Scal­ing Law of Mix­ture-of-Experts. arXiv:2509.23678.
464. Changxin Tian, Kun­long Chen, Jia Liu et al. (2025). Towards Greater Lever­age: Scal­ing Laws for Effi­cient Mix­ture-of-Experts Lan­guage Models. arXiv:2507.17702.
465. Elias Fran­tar, Carlos Riquelme, Neil Houlsby, Dan Alis­tarh and Utku Evci (2023). Scal­ing Laws for Sparsely-Con­nected Foun­da­tion Models. arXiv:2309.08520.
466. Xingwu Sun, Shuaipeng Li, Ruob­ing Xie et al. (2025). Scal­ing Laws for Float­ing Point Quan­ti­za­tion Train­ing. arXiv:2501.02423.
467. Mengzhao Chen, Chaoyi Zhang, Jing Liu et al. (2025). Scal­ing Law for Quan­ti­za­tion-Aware Train­ing. arXiv:2505.14302.
468. Tim Dettmers and Luke Zettle­moyer (2022). The case for 4-bit pre­ci­sion: k-bit Infer­ence Scal­ing Laws. arXiv:2212.09720.
469. Xu Ouyang, Tao Ge, Thomas Hartvigsen et al. (2024). Low-Bit Quan­ti­za­tion Favors Under­trained LLMs: Scal­ing Laws for Quan­tized LLMs with 100T Train­ing Tokens. arXiv:2411.17691.
470. Moux­i­ang Chen, Binyuan Hui, Zeyu Cui et al. (2025). Par­al­lel Scal­ing Law for Lan­guage Models. arXiv:2505.10475.
471. Dan Bus­bridge, Amitis Shi­dani, Floris Weers et al. (2025). Dis­til­la­tion Scal­ing Laws. arXiv:2502.08606.
472. OpenAI, Josh Achiam, Steven Adler et al. (2023). GPT-4 Tech­ni­cal Report. arXiv:2303.08774.
473. Akshita Bhagia, Jiacheng Liu, Alexan­der Wettig et al. (2024). Estab­lish­ing Task Scal­ing Laws via Com­pute-Effi­cient Model Lad­ders. arXiv:2412.04403.
474. Nicholas Lourie, Michael Y. Hu and Kyunghyun Cho (2025). Scal­ing Laws Are Unre­li­able for Down­stream Tasks: A Real­ity Check. arXiv:2507.00885.
475. Rylan Scha­ef­fer, Hailey Schoelkopf, Brando Miranda et al. (2024). Why Has Pre­dict­ing Down­stream Capa­bil­i­ties of Fron­tier AI Models with Scale Remained Elu­sive?. arXiv:2406.04391.
476. Jason Wei, Yi Tay, Rishi Bom­masani et al. (2022). Emer­gent Abil­i­ties of Large Lan­guage Models. arXiv:2206.07682.
477. Rylan Scha­ef­fer, Brando Miranda and Sanmi Koyejo (2023). Are Emer­gent Abil­i­ties of Large Lan­guage Models a Mirage?. arXiv:2304.15004.
478. Zhengx­iao Du, Aohan Zeng, Yuxiao Dong and Jie Tang (2024). Under­stand­ing Emer­gent Abil­i­ties of Lan­guage Models from the Loss Per­spec­tive. arXiv:2403.15796.
479. Bradley Brown, Jordan Juravsky, Ryan Ehrlich et al. (2024). Large Lan­guage Mon­keys: Scal­ing Infer­ence Com­pute with Repeated Sam­pling. arXiv:2407.21787.
480. Rylan Scha­ef­fer, Joshua Kazdan, John Hughes et al. (2025). How Do Large Lan­guage Mon­keys Get Their Power (Laws)?. arXiv:2502.17578.
481. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck and Yiming Yang (2024). Infer­ence Scal­ing Laws: An Empir­i­cal Anal­y­sis of Com­pute-Opti­mal Infer­ence for Prob­lem-Solv­ing with Lan­guage Models. arXiv:2408.00724.
482. Niklas Muen­nighoff, Zitong Yang, Weijia Shi et al. (2025). s1: Simple test-time scal­ing. arXiv:2501.19393.
483. Devvrit Khatri, Lovish Madaan, Rishabh Tiwari et al. (2025). The Art of Scal­ing Rein­force­ment Learn­ing Com­pute for LLMs. arXiv:2510.13786.
484. Yang Yue, Zhiqi Chen, Rui Lu et al. (2025). Does Rein­force­ment Learn­ing Really Incen­tivize Rea­son­ing Capac­ity in LLMs Beyond the Base Model?. arXiv:2504.13837.
485. Mingjie Liu, Shizhe Diao, Ximing Lu et al. (2025). ProRL: Pro­longed Rein­force­ment Learn­ing Expands Rea­son­ing Bound­aries in Large Lan­guage Models. arXiv:2505.24864.
486. Thomas Kwa, Ben West, Joel Becker et al. (2025). Mea­sur­ing AI Abil­ity to Com­plete Long Soft­ware Tasks. arXiv:2503.14499.
487. METR (2026). Time Hori­zon 1.1. METR blog.
488. Karl Cobbe, Vineet Kosaraju, Moham­mad Bavar­ian et al. (2021). Train­ing Ver­i­fiers to Solve Math Word Prob­lems. arXiv:2110.14168.
489. Chengy­ing Tu, Xuemiao Zhang, Rongx­i­ang Weng et al. (2025). A Survey on LLM Mid-Train­ing. arXiv:2510.23081.
490. Kaix­i­ang Mo, Yuxin Shi, Weiwei Weng et al. (2025). Mid-Train­ing of Large Lan­guage Models: A Survey. arXiv:2510.06826.
491. Elie Bak­ouch, Carlos Miguel Patiño, Anton Lozhkov et al. (2025). SmolLM3: smol, mul­ti­lin­gual, long-con­text rea­soner. Hug­ging Face blog.
492. Zengzhi Wang, Fan Zhou, Xue­feng Li and Pengfei Liu (2025). OctoThinker: Mid-train­ing Incen­tivizes Rein­force­ment Learn­ing Scal­ing. arXiv:2506.20512.
493. Soumye Sing­hal, Joseph Jen­nings, Ka Shun Shum et al. (2026). Prim­ing Base Models for Agen­tic Post-Train­ing: An Open Mid­train­ing Recipe. arXiv:2610.22019.
494. Meta AI (2024). Llama 3.2: Rev­o­lu­tion­iz­ing edge AI and vision with open, cus­tom­iz­a­ble models. Meta AI blog.
495. Saurav Muralid­ha­ran, Sharath Turu­vekere Sreeni­vas, Ravi­raj Joshi et al. (2024). Com­pact Lan­guage Models via Prun­ing and Knowl­edge Dis­til­la­tion. arXiv:2407.14679.
496. Steven Feng, Shri­mai Prab­hu­moye, Kezhi Kong et al. (2024). Max­i­mize Your Data's Poten­tial: Enhanc­ing LLM Accu­racy with Two-Phase Pre­train­ing. arXiv:2412.15285.
497. Yuling Gu, Oyvind Tafjord, Bailey Kuehl et al. (2024). OLMES: A Stan­dard for Lan­guage Model Eval­u­a­tions. arXiv:2406.08446.
498. Dan Hendrycks, Collin Burns, Steven Basart et al. (2020). Mea­sur­ing Mas­sive Mul­ti­task Lan­guage Under­stand­ing. arXiv:2009.03300.
499. Ricardo Dominguez-Olmedo, Flo­rian E. Dorner and Moritz Hardt (2024). Train­ing on the Test Task Con­founds Eval­u­a­tion and Emer­gence. arXiv:2407.07890.
500. Bharat Runwal, Ashish Agrawal, Anurag Roy and Rameswar Panda (2026). PRISM: Demys­ti­fy­ing Reten­tion and Inter­ac­tion in Mid-Train­ing. arXiv:2603.17074.
501. Syeda Nahida Akter, Shri­mai Prab­hu­moye, Eric Nyberg et al. (2025). Front-Load­ing Rea­son­ing: The Syn­ergy between Pre­train­ing and Post-Train­ing Data. arXiv:2510.03264.
502. Emmy Liu, Graham Neubig and Chenyan Xiong (2025). Mid­train­ing Bridges Pre­train­ing and Post­train­ing Dis­tri­bu­tions. arXiv:2510.14865.
503. Syeda Nahida Akter, Shri­mai Prab­hu­moye, John Kamalu et al. (2024). MIND: Math Informed syNthetic Dia­logues for Pre­train­ing LLMs. arXiv:2410.12881.
504. Shuo Yang, Wei-Lin Chiang, Lian­min Zheng, Joseph E. Gon­za­lez and Ion Stoica (2023). Rethink­ing Bench­mark and Con­tam­i­na­tion for Lan­guage Models with Rephrased Sam­ples. arXiv:2311.04850.
505. Wenhan Xiong, Jingyu Liu, Igor Moly­bog et al. (2023). Effec­tive Long-Con­text Scal­ing of Foun­da­tion Models. arXiv:2309.16039.
506. Yao Fu, Rameswar Panda, Xinyao Niu et al. (2024). Data Engi­neer­ing for Scal­ing Lan­guage Models to 128K Con­text. arXiv:2402.10171.
507. Bowen Peng, Jef­frey Ques­nelle, Honglu Fan and Enrico Ship­pole (2023). YaRN: Effi­cient Con­text Window Exten­sion of Large Lan­guage Models. arXiv:2309.00071.
508. Xiao­ran Liu, Hang Yan, Shuo Zhang et al. (2023). Scal­ing Laws of RoPE-based Extrap­o­la­tion. arXiv:2310.05209.
509. Shouyuan Chen, Sher­man Wong, Liangjian Chen and Yuan­dong Tian (2023). Extend­ing Con­text Window of Large Lan­guage Models via Posi­tional Inter­po­la­tion. arXiv:2306.15595.
510. Hug­ging Face (2026). trans­form­ers/src/trans­form­ers/mod­el­ing_rope_utils.py. GitHub repos­i­tory (hug­ging­face/trans­form­ers).
511. Xin Men, Mingyu Xu, Bingn­ing Wang et al. (2024). Base of RoPE Bounds Con­text Length. arXiv:2405.14591.
512. Tianyu Gao, Alexan­der Wettig, Howard Yen and Danqi Chen (2024). How to Train Long-Con­text Lan­guage Models (Effec­tively). arXiv:2410.02660.
513. Amirhos­sein Kazem­ne­jad, Inkit Padhi, Karthikeyan Nate­san Rama­murthy, Payel Das and Siva Reddy (2023). The Impact of Posi­tional Encod­ing on Length Gen­er­al­iza­tion in Trans­form­ers. arXiv:2305.19466.
514. Yiran Ding, Li Lyna Zhang, Chen­gruidong Zhang et al. (2024). LongRoPE: Extend­ing LLM Con­text Window Beyond 2 Mil­lion Tokens. arXiv:2402.13753.
515. Bap­tiste Rozière, Jonas Gehring, Fabian Gloeckle et al. (2023). Code Llama: Open Foun­da­tion Models for Code. arXiv:2308.12950.
516. An Yang, Bowen Yu, Chengyuan Li et al. (2025). Qwen2.5⁠–⁠1M Tech­ni­cal Report. arXiv:2501.15383.
517. Ziqing Qiao, Xiaoyue Xu, Sikui Zhang et al. (2026). Don't Stop Long-Con­text Con­tin­ual Pre-Train­ing Too Early: An Empir­i­cal Scal­ing Law. arXiv:2610.29352.
518. Greg Kam­radt (2023). GitHub - gkam­radt/needle-in-a-haystack: Doing simple retrieval from LLM models at var­i­ous con­text lengths to mea­sure accu­racy. GitHub repos­i­tory.
519. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman et al. (2024). RULER: What's the Real Con­text Size of Your Long-Con­text Lan­guage Models?. arXiv:2404.06654.
520. Ali Modar­ressi, Hanieh Deil­am­salehy, Franck Der­non­court et al. (2025). NoLiMa: Long-Con­text Eval­u­a­tion Beyond Lit­eral Match­ing. arXiv:2502.05167.
521. Yuri Kura­tov, Aydar Bula­tov, Petr Anokhin et al. (2024). BABILong: Test­ing the Limits of LLMs with Long Con­text Rea­son­ing-in-a-Haystack. arXiv:2406.10149.
522. Nelson F. Liu, Kevin Lin, John Hewitt et al. (2023). Lost in the Middle: How Lan­guage Models Use Long Con­texts. arXiv:2307.03172.
523. Howard Yen, Tianyu Gao, Minmin Hou et al. (2024). HELMET: How to Eval­u­ate Long-Con­text Lan­guage Models Effec­tively and Thor­oughly. arXiv:2410.02694.
524. Yi Lu, Jing Nathan Yan, Songlin Yang et al. (2024). A Con­trolled Study on Long Con­text Exten­sion and Gen­er­al­iza­tion in LLMs. arXiv:2409.12181.
525. Lizhe Fang, Yifei Wang, Zhaoyang Liu et al. (2024). What is Wrong with Per­plex­ity for Long-con­text Lan­guage Mod­el­ing?. arXiv:2410.23771.
526. Kshi­tij Gupta, Ben­jamin Thérien, Adam Ibrahim et al. (2023). Con­tin­ual Pre-Train­ing of Large Lan­guage Models: How to (re)warm your model?. arXiv:2308.04014.
527. Suchin Guru­ran­gan, Ana Marasović, Swabha Swayamdipta et al. (2020). Don't Stop Pre­train­ing: Adapt Lan­guage Models to Domains and Tasks. arXiv:2004.10964.
528. Jupin­der Parmar, Sanjev Satheesh, Mostofa Pat­wary, Moham­mad Shoeybi and Bryan Catan­zaro (2024). Reuse, Don't Retrain: A Recipe for Con­tin­ued Pre­train­ing of Lan­guage Models. arXiv:2407.07263.
529. James Kirk­patrick, Razvan Pas­canu, Neil Rabi­nowitz et al. (2016). Over­com­ing cat­a­strophic for­get­ting in neural net­works. arXiv:1612.00796.
530. Dan Bider­man, Jacob Portes, Jose Javier Gon­za­lez Ortiz et al. (2024). LoRA Learns Less and For­gets Less. arXiv:2405.09673.
531. Junhao Zheng, Xidi Cai, Shengjie Qiu and Qianli Ma (2025). Spu­ri­ous For­get­ting in Con­tin­ual Learn­ing of Lan­guage Models. arXiv:2501.13453.
532. Jonghyun Han, Younghoon Song and Jongy­oul Park (2026). A Deaf­en­ing Silence: Cat­a­strophic For­get­ting Lives in the Output Embed­dings of Tokens the Data Never Speaks. arXiv:2610.09835.
533. Zhangir Azer­bayev, Hailey Schoelkopf, Keiran Paster et al. (2023). Llemma: An Open Lan­guage Model For Math­e­mat­ics. arXiv:2310.10631.
534. DeepSeek-AI, Qihao Zhu, Daya Guo et al. (2024). DeepSeek-Coder-V2: Break­ing the Bar­rier of Closed-Source Models in Code Intel­li­gence. arXiv:2406.11931.
535. Kazuki Fujii, Taishi Naka­mura, Mengsay Loem et al. (2024). Con­tin­ual Pre-Train­ing for Cross-Lin­gual LLM Adap­ta­tion: Enhanc­ing Japa­nese Lan­guage Capa­bil­i­ties. arXiv:2404.17790.
536. Zitong Yang, Neil Band, Shuang­ping Li, Emmanuel Candès and Tat­sunori Hashimoto (2024). Syn­thetic con­tin­ued pre­train­ing. arXiv:2409.07431.
537. Yiming Cui, Ziqing Yang and Xin Yao (2023). Effi­cient and Effec­tive Text Encod­ing for Chi­nese LLaMA and Alpaca. arXiv:2304.08177.
538. Geof­frey Hinton, Oriol Vinyals and Jeff Dean (2015). Dis­till­ing the Knowl­edge in a Neural Net­work. arXiv:1503.02531.
539. Yuxian Gu, Li Dong, Furu Wei and Minlie Huang (2023). MiniLLM: On-Policy Dis­til­la­tion of Large Lan­guage Models. arXiv:2306.08543.
540. Rishabh Agar­wal, Nino Vieil­lard, Yongchao Zhou et al. (2023). On-Policy Dis­til­la­tion of Lan­guage Models: Learn­ing from Self-Gen­er­ated Mis­takes. arXiv:2306.13649.
541. Yoon Kim and Alexan­der M. Rush (2016). Sequence-Level Knowl­edge Dis­til­la­tion. arXiv:1606.07947.
542. Meta (2024). Llama 3.2 Model Card. Model card (GitHub).
543. Taim­ing Lu and Zhuang Liu (2026). Strong Teacher Not Needed? On Dis­til­la­tion in LLM Pre­train­ing. arXiv:2605.23857.
544. Sharath Turu­vekere Sreeni­vas, Saurav Muralid­ha­ran, Ravi­raj Joshi et al. (2024). LLM Prun­ing and Dis­til­la­tion in Prac­tice: The Mini­tron Approach. arXiv:2408.11796.
545. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng and Danqi Chen (2023). Sheared LLaMA: Accel­er­at­ing Lan­guage Model Pre-train­ing via Struc­tured Prun­ing. arXiv:2310.06694.
546. Shijie Wu, Ozan Irsoy, Steven Lu et al. (2023). BloombergGPT: A Large Lan­guage Model for Finance. arXiv:2303.17564.
547. Nathan Lam­bert (2025). Rein­force­ment Learn­ing from Human Feed­back. arXiv:2504.12501.
548. Chunt­ing Zhou, Pengfei Liu, Puxin Xu et al. (2023). LIMA: Less Is More for Align­ment. arXiv:2305.11206.
549. Bill Yuchen Lin, Abhi­lasha Ravichan­der, Ximing Lu et al. (2023). The Unlock­ing Spell on Base LLMs: Rethink­ing Align­ment via In-Con­text Learn­ing. arXiv:2312.01552.
550. Nathan Lam­bert, Jacob Mor­ri­son, Valentina Pyatkin et al. (2024). Tulu 3: Push­ing Fron­tiers in Open Lan­guage Model Post-Train­ing. arXiv:2411.15124.
551. Jason Wei, Maarten Bosma, Vin­cent Y. Zhao et al. (2021). Fine­tuned Lan­guage Models Are Zero-Shot Learn­ers. arXiv:2109.01652.
552. Victor Sanh, Albert Webson, Colin Raffel et al. (2021). Mul­ti­task Prompted Train­ing Enables Zero-Shot Task Gen­er­al­iza­tion. arXiv:2110.08207.
553. Yizhong Wang, Swa­roop Mishra, Pegah Alipoor­mo­labashi et al. (2022). Super-NaturalInstructions: Gen­er­al­iza­tion via Declar­a­tive Instruc­tions on 1600+ NLP Tasks. arXiv:2204.07705.
554. Hyung Won Chung, Le Hou, Shayne Long­pre et al. (2022). Scal­ing Instruc­tion-Fine­tuned Lan­guage Models. arXiv:2210.11416.
555. Shayne Long­pre, Le Hou, Tu Vu et al. (2023). The Flan Col­lec­tion: Design­ing Data and Meth­ods for Effec­tive Instruc­tion Tuning. arXiv:2301.13688.
556. Arnav Gudibande, Eric Wal­lace, Char­lie Snell et al. (2023). The False Prom­ise of Imi­tat­ing Pro­pri­etary LLMs. arXiv:2305.15717.
557. Yizhong Wang, Yeganeh Kordi, Swa­roop Mishra et al. (2022). Self-Instruct: Align­ing Lan­guage Models with Self-Gen­er­ated Instruc­tions. arXiv:2212.10560.
558. Wei-Lin Chiang, Zhuo­han Li, Zi Lin et al. (2023). Vicuna: An Open-Source Chat­bot Impress­ing GPT-4 with 90%* ChatGPT Qual­ity. LMSYS Org blog.
559. Meta (2024). Llama 3 | Model Cards and Prompt for­mats. Llama doc­u­men­ta­tion.
560. Hug­ging Face (2025). Chat tem­plates. Hug­ging Face Trans­form­ers doc­u­men­ta­tion.
561. Zhengyan Shi, Adam X. Yang, Bin Wu et al. (2024). Instruc­tion Tuning With Loss Over Instruc­tions. arXiv:2405.14394.
562. Lewis Tun­stall, Edward Beech­ing, Nathan Lam­bert et al. (2023). Zephyr: Direct Dis­til­la­tion of LM Align­ment. arXiv:2310.16944.
563. Neel Jain, Ping-yeh Chiang, Yuxin Wen et al. (2023). NEFTune: Noisy Embed­dings Improve Instruc­tion Fine­tun­ing. arXiv:2310.05914.
564. Idan Shen­feld, Jyoth­ish Pari and Pulkit Agrawal (2025). RL's Razor: Why Online Rein­force­ment Learn­ing For­gets Less. arXiv:2509.04259.
565. Edward J. Hu, Yelong Shen, Phillip Wallis et al. (2021). LoRA: Low-Rank Adap­ta­tion of Large Lan­guage Models. arXiv:2106.09685.
566. John Schul­man and Think­ing Machines Lab (2025). LoRA With­out Regret. Think­ing Machines Lab: Con­nec­tion­ism (blog).
567. Damjan Kala­jdzievski (2023). A Rank Sta­bi­liza­tion Scal­ing Factor for Fine-Tuning with LoRA. arXiv:2312.03732.
568. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin et al. (2024). DoRA: Weight-Decom­posed Low-Rank Adap­ta­tion. arXiv:2402.09353.
569. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts­man et al. (2022). Edit­ing Models with Task Arith­metic. arXiv:2212.04089.
570. Pra­teek Yadav, Derek Tam, Leshem Choshen, Colin Raffel and Mohit Bansal (2023). TIES-Merg­ing: Resolv­ing Inter­fer­ence When Merg­ing Models. arXiv:2306.01708.
571. Paul Chris­tiano, Jan Leike, Tom B. Brown et al. (2017). Deep rein­force­ment learn­ing from human pref­er­ences. arXiv:1706.03741.
572. Nisan Sti­en­non, Long Ouyang, Jeff Wu et al. (2020). Learn­ing to sum­ma­rize from human feed­back. arXiv:2009.01325.
573. Amanda Askell, Yuntao Bai, Anna Chen et al. (2021). A Gen­eral Lan­guage Assis­tant as a Lab­o­ra­tory for Align­ment. arXiv:2112.00861.
574. Ralph Allan Bradley and Milton E. Terry (1952). Rank Anal­y­sis of Incom­plete Block Designs: I. The Method of Paired Com­par­isons. Biometrika.
575. Leo Gao, John Schul­man and Jacob Hilton (2022). Scal­ing Laws for Reward Model Overop­ti­miza­tion. arXiv:2210.10760.
576. Nathan Lam­bert, Valentina Pyatkin, Jacob Mor­ri­son et al. (2024). RewardBench: Eval­u­at­ing Reward Models for Lan­guage Mod­el­ing. arXiv:2403.13787.
577. Zhilin Wang, Yi Dong, Olivier Delal­leau et al. (2024). HelpSteer2: Open-source dataset for train­ing top-per­form­ing reward models. arXiv:2406.08673.
578. Noam Razin, Zixuan Wang, Hubert Strauss et al. (2025). What Makes a Reward Model a Good Teacher? An Opti­miza­tion Per­spec­tive. arXiv:2503.15477.
579. Yanjun Chen, Dawei Zhu, Yirong Sun et al. (2024). The Accu­racy Para­dox in RLHF: When Better Reward Models Don't Yield Better Lan­guage Models. arXiv:2410.06554.
580. Yuntao Bai, Saurav Kada­vath, Sandi­pan Kundu et al. (2022). Con­sti­tu­tional AI: Harm­less­ness from AI Feed­back. arXiv:2212.08073.
581. Har­ri­son Lee, Samrat Phatale, Hassan Man­soor et al. (2023). RLAIF vs. RLHF: Scal­ing Rein­force­ment Learn­ing from Human Feed­back with AI Feed­back. arXiv:2309.00267.
582. Lunjun Zhang, Arian Hos­seini, Hritik Bansal et al. (2024). Gen­er­a­tive Ver­i­fiers: Reward Mod­el­ing as Next-Token Pre­dic­tion. arXiv:2408.15240.
583. Dakota Mahan, Duy Van Phung, Rafael Rafailov et al. (2024). Gen­er­a­tive Reward Models. arXiv:2410.12832.
584. Anisha Gunjal, Anthony Wang, Elaine Lau et al. (2025). Rubrics as Rewards: Rein­force­ment Learn­ing Beyond Ver­i­fi­able Domains. arXiv:2507.17746.
585. Vijay Viswanathan, Yan­chao Sun, Shuang Ma et al. (2025). Check­lists Are Better Than Reward Models For Align­ing Lan­guage Models. arXiv:2507.18624.
586. Rafael Rafailov, Archit Sharma, Eric Mitchell et al. (2023). Direct Pref­er­ence Opti­miza­tion: Your Lan­guage Model is Secretly a Reward Model. arXiv:2305.18290.
587. Ahmad Beirami, Alekh Agar­wal, Jonathan Berant et al. (2024). The­o­ret­i­cal guar­an­tees on the best-of-n align­ment policy. arXiv:2401.01879.
588. Hanze Dong, Wei Xiong, Deep­an­shu Goyal et al. (2023). RAFT: Reward rAnked FineTuning for Gen­er­a­tive Foun­da­tion Model Align­ment. arXiv:2304.06767.
589. Caglar Gul­cehre, Tom Le Paine, Sri­vat­san Srini­vasan et al. (2023). Rein­forced Self-Train­ing (ReST) for Lan­guage Mod­el­ing. arXiv:2308.08998.
590. John Schul­man (2020). Approx­i­mat­ing KL Diver­gence. Blog post.
591. John Schul­man, Filip Wolski, Pra­fulla Dhari­wal, Alec Rad­ford and Oleg Klimov (2017). Prox­i­mal Policy Opti­miza­tion Algo­rithms. arXiv:1707.06347.
592. John Schul­man, Philipp Moritz, Sergey Levine, Michael Jordan and Pieter Abbeel (2015). High-Dimen­sional Con­tin­u­ous Con­trol Using Gen­er­al­ized Advan­tage Esti­ma­tion. arXiv:1506.02438.
593. Shengyi Huang, Michael Noukhovitch, Arian Hos­seini et al. (2024). The N+ Imple­men­ta­tion Details of RLHF with PPO: A Case Study on TL;DR Sum­ma­riza­tion. arXiv:2403.17031.
594. Daniel M. Ziegler, Nisan Sti­en­non, Jef­frey Wu et al. (2019). Fine-Tuning Lan­guage Models from Human Pref­er­ences. arXiv:1909.08593.
595. Yuntao Bai, Andy Jones, Kamal Ndousse et al. (2022). Train­ing a Help­ful and Harm­less Assis­tant with Rein­force­ment Learn­ing from Human Feed­back. arXiv:2204.05862.
596. Rui Zheng, Shihan Dou, Songyang Gao et al. (2023). Secrets of RLHF in Large Lan­guage Models Part I: PPO. arXiv:2307.04964.
597. Lorenz Wolf, Robert Kirk and Mirco Musolesi (2025). Reward Model Overop­ti­mi­sa­tion in Iter­ated RLHF. arXiv:2505.18126.
598. Thomas Coste, Usman Anwar, Robert Kirk and David Krueger (2023). Reward Model Ensem­bles Help Mit­i­gate Overop­ti­miza­tion. arXiv:2310.02743.
599. Jacob Eisen­stein, Chirag Nagpal, Alekh Agar­wal et al. (2023). Help­ing or Herd­ing? Reward Model Ensem­bles Mit­i­gate but do not Elim­i­nate Reward Hack­ing. arXiv:2312.09244.
600. Alexan­dre Ramé, Nino Vieil­lard, Léonard Hussenot et al. (2024). WARM: On the Ben­e­fits of Weight Aver­aged Reward Models. arXiv:2401.12187.
601. Rafael Rafailov, Yaswanth Chit­tepu, Ryan Park et al. (2024). Scal­ing Laws for Reward Model Overop­ti­miza­tion in Direct Align­ment Algo­rithms. arXiv:2406.02900.
602. Prasann Sing­hal, Tanya Goyal, Jiacheng Xu and Greg Dur­rett (2023). A Long Way to Go: Inves­ti­gat­ing Length Cor­re­la­tions in RLHF. arXiv:2310.03716.
603. Yann Dubois, Balázs Galam­bosi, Percy Liang and Tat­sunori B. Hashimoto (2024). Length-Con­trolled AlpacaEval: A Simple Way to Debias Auto­matic Eval­u­a­tors. arXiv:2404.04475.
604. Lichang Chen, Chen Zhu, Davit Soselia et al. (2024). ODIN: Dis­en­tan­gled Reward Mit­i­gates Hack­ing in RLHF. arXiv:2402.07319.
605. Mri­nank Sharma, Meg Tong, Tomasz Korbak et al. (2023). Towards Under­stand­ing Syco­phancy in Lan­guage Models. arXiv:2310.13548.
606. Jiaxin Wen, Ruiqi Zhong, Akbir Khan et al. (2024). Lan­guage Models Learn to Mis­lead Humans via RLHF. arXiv:2409.12822.
607. OpenAI (2025). Expand­ing on what we missed with syco­phancy. OpenAI blog.
608. OpenAI (2025). Syco­phancy in GPT-4o: what hap­pened and what we’re doing about it. OpenAI blog.
609. Robert Kirk, Ishita Medi­ratta, Christo­foros Nalm­pan­tis et al. (2023). Under­stand­ing the Effects of RLHF on LLM Gen­er­al­i­sa­tion and Diver­sity. arXiv:2310.06452.
610. Monte MacDiarmid, Ben­jamin Wright, Jonathan Uesato et al. (2025). Nat­u­ral Emer­gent Mis­align­ment from Reward Hack­ing in Pro­duc­tion RL. arXiv:2511.18397.
611. Kawin Etha­yarajh, Winnie Xu, Niklas Muen­nighoff, Dan Juraf­sky and Douwe Kiela (2024). KTO: Model Align­ment as Prospect The­o­retic Opti­miza­tion. arXiv:2402.01306.
612. Moham­mad Ghesh­laghi Azar, Mark Row­land, Bilal Piot et al. (2023). A Gen­eral The­o­ret­i­cal Par­a­digm to Under­stand Learn­ing from Human Pref­er­ences. arXiv:2310.12036.
613. Yu Meng, Mengzhou Xia and Danqi Chen (2024). SimPO: Simple Pref­er­ence Opti­miza­tion with a Ref­er­ence-Free Reward. arXiv:2405.14734.
614. Jiwoo Hong, Noah Lee and James Thorne (2024). ORPO: Mono­lithic Pref­er­ence Opti­miza­tion with­out Ref­er­ence Model. arXiv:2403.07691.
615. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho et al. (2024). Iter­a­tive Rea­son­ing Pref­er­ence Opti­miza­tion. arXiv:2404.19733.
616. Ryan Park, Rafael Rafailov, Ste­fano Ermon and Chelsea Finn (2024). Dis­en­tan­gling Length from Qual­ity in Direct Pref­er­ence Opti­miza­tion. arXiv:2403.19159.
617. Arka Pal, Deep Karkha­nis, Samuel Dooley et al. (2024). Smaug: Fixing Fail­ure Modes of Pref­er­ence Opti­mi­sa­tion with DPO-Pos­i­tive. arXiv:2402.13228.
618. Noam Razin, Sad­hika Mal­ladi, Adithya Bhaskar et al. (2024). Unin­ten­tional Unalign­ment: Like­li­hood Dis­place­ment in Direct Pref­er­ence Opti­miza­tion. arXiv:2410.08847.
619. Shusheng Xu, Wei Fu, Jiax­uan Gao et al. (2024). Is DPO Supe­rior to PPO for LLM Align­ment? A Com­pre­hen­sive Study. arXiv:2404.10719.
620. Shang­min Guo, Biao Zhang, Tian­lin Liu et al. (2024). Direct Lan­guage Model Align­ment from Online AI Feed­back. arXiv:2402.04792.
621. Yunhao Tang, Daniel Zhao­han Guo, Zeyu Zheng et al. (2024). Under­stand­ing the per­for­mance gap between online and off­line align­ment algo­rithms. arXiv:2405.08448.
622. Jack Lan­chantin, Angel­ica Chen, Janice Lan et al. (2025). Bridg­ing Off­line and Online Rein­force­ment Learn­ing for LLMs. arXiv:2506.21495.
623. Hamish Ivison, Yizhong Wang, Jiacheng Liu et al. (2024). Unpack­ing DPO and PPO: Dis­en­tan­gling Best Prac­tices for Learn­ing from Pref­er­ence Feed­back. arXiv:2406.09279.
624. Yao Xiao, Hai Ye, Linyao Chen et al. (2025). Find­ing the Sweet Spot: Pref­er­ence Data Con­struc­tion for Scal­ing Pref­er­ence Opti­miza­tion. arXiv:2502.16825.
625. Taey­oun Kim and Aviral Kumar (2026). Address­ing Over-Refusal in LLMs with Com­pet­ing Rewards. arXiv:2606.31748.
626. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen et al. (2023). XSTest: A Test Suite for Iden­ti­fy­ing Exag­ger­ated Safety Behav­iours in Large Lan­guage Models. arXiv:2308.01263.
627. Justin Cui, Wei-Lin Chiang, Ion Stoica and Cho-Jui Hsieh (2024). OR-Bench: An Over-Refusal Bench­mark for Large Lan­guage Models. arXiv:2405.20947.
628. Josef Dai, Xuehai Pan, Ruiyang Sun et al. (2023). Safe RLHF: Safe Rein­force­ment Learn­ing from Human Feed­back. arXiv:2310.12773.
629. Deep Gan­guli, Liane Lovitt, Jack­son Kernion et al. (2022). Red Team­ing Lan­guage Models to Reduce Harms: Meth­ods, Scal­ing Behav­iors, and Lessons Learned. arXiv:2209.07858.
630. Ethan Perez, Saf­fron Huang, Fran­cis Song et al. (2022). Red Team­ing Lan­guage Models with Lan­guage Models. arXiv:2202.03286.
631. Tong Mu, Alec Helyar, Johannes Hei­decke et al. (2024). Rule Based Rewards for Lan­guage Model Safety. arXiv:2411.01111.
632. Melody Y. Guan, Manas Joglekar, Eric Wal­lace et al. (2024). Delib­er­a­tive Align­ment: Rea­son­ing Enables Safer Lan­guage Models. arXiv:2412.16339.
633. Eric Wal­lace, Kai Xiao, Reimar Leike et al. (2024). The Instruc­tion Hier­ar­chy: Train­ing LLMs to Pri­or­i­tize Priv­i­leged Instruc­tions. arXiv:2404.13208.
634. OpenAI (2025). OpenAI Model Spec. OpenAI.
635. Andy Arditi, Oscar Obeso, Aaquib Syed et al. (2024). Refusal in Lan­guage Models Is Medi­ated by a Single Direc­tion. arXiv:2406.11717.
636. Anthropic (2024). Claude's char­ac­ter. Anthropic research blog.
637. Sharan Maiya, Hen­ning Bartsch, Nathan Lam­bert and Evan Hub­inger (2025). Open Char­ac­ter Train­ing: Shap­ing the Per­sona of AI Assis­tants through Con­sti­tu­tional AI. arXiv:2511.01689.
638. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans and Jack Lind­sey (2025). Per­sona Vec­tors: Mon­i­tor­ing and Con­trol­ling Char­ac­ter Traits in Lan­guage Models. arXiv:2507.21509.
639. Daya Guo, Dejian Yang, Haowei Zhang et al. (2025). DeepSeek-R1 incen­tivizes rea­son­ing in LLMs through rein­force­ment learn­ing. Nature.
640. Qiying Yu, Zheng Zhang, Ruofei Zhu et al. (2025). DAPO: An Open-Source LLM Rein­force­ment Learn­ing System at Scale. arXiv:2503.14476.
641. Hug­ging Face (2025). hug­ging­face/Math-Verify. GitHub repos­i­tory.
642. Hunter Light­man, Vineet Kosaraju, Yura Burda et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
643. Jonathan Uesato, Nate Kush­man, Ramana Kumar et al. (2022). Solv­ing math word prob­lems with process- and out­come-based feed­back. arXiv:2211.14275.
644. Peiyi Wang, Lei Li, Zhi­hong Shao et al. (2023). Math-Shep­herd: Verify and Rein­force LLMs Step-by-step with­out Human Anno­ta­tions. arXiv:2312.08935.
645. Amrith Setlur, Chirag Nagpal, Adam Fisch et al. (2024). Reward­ing Progress: Scal­ing Auto­mated Process Ver­i­fiers for LLM Rea­son­ing. arXiv:2410.08146.
646. Chujie Zheng, Zhenru Zhang, Beichen Zhang et al. (2024). ProcessBench: Iden­ti­fy­ing Process Errors in Math­e­mat­i­cal Rea­son­ing. arXiv:2412.06559.
647. Zhenru Zhang, Chujie Zheng, Yangzhen Wu et al. (2025). The Lessons of Devel­op­ing Process Reward Models in Math­e­mat­i­cal Rea­son­ing. arXiv:2501.07301.
648. Kimi Team, Angang Du, Bofei Gao et al. (2025). Kimi k1.5: Scal­ing Rein­force­ment Learn­ing with LLMs. arXiv:2501.12599.
649. Rulin Shao, Shuyue Stella Li, Rui Xin et al. (2025). Spu­ri­ous Rewards: Rethink­ing Train­ing Sig­nals in RLVR. arXiv:2506.10947.
650. Ronald J. Williams (1992). Simple sta­tis­ti­cal gra­di­ent-fol­low­ing algo­rithms for con­nec­tion­ist rein­force­ment learn­ing. Machine Learn­ing.
651. Ziniu Li, Tian Xu, Yushun Zhang et al. (2023). ReMax: A Simple, Effec­tive, and Effi­cient Rein­force­ment Learn­ing Method for Align­ing Large Lan­guage Models. arXiv:2310.10505.
652. Arash Ahma­dian, Chris Cremer, Matthias Gallé et al. (2024). Back to Basics: Revis­it­ing REINFORCE Style Opti­miza­tion for Learn­ing from Human Feed­back in LLMs. arXiv:2402.14740.
653. Eric Zelik­man, Yuhuai Wu, Jesse Mu and Noah D. Good­man (2022). STaR: Boot­strap­ping Rea­son­ing With Rea­son­ing. arXiv:2203.14465.
654. Avi Singh, John D. Co-Reyes, Rishabh Agar­wal et al. (2023). Beyond Human Data: Scal­ing Self-Train­ing for Prob­lem-Solv­ing with Lan­guage Models. arXiv:2312.06585.
655. Lak­shya Chaudhry (2026). Is it RL or Data? A Con­trolled Com­par­i­son of GRPO and Rejec­tion-Sam­pling Fine-Tuning. arXiv:2610.25812.
656. MiniMax, Aili Chen, Aonian Li et al. (2025). MiniMax-M1: Scal­ing Test-Time Com­pute Effi­ciently with Light­ning Atten­tion. arXiv:2506.13585.
657. Yu Yue, Yufeng Yuan, Qiying Yu et al. (2025). VAPO: Effi­cient and Reli­able Rein­force­ment Learn­ing for Advanced Rea­son­ing Tasks. arXiv:2504.05118.
658. Jingcheng Hu, Yinmin Zhang, Qi Han et al. (2025). Open-Rea­soner-Zero: An Open Source Approach to Scal­ing Up Rein­force­ment Learn­ing on the Base Model. arXiv:2503.24290.
659. Logan Engstrom, Andrew Ilyas, Shibani San­turkar et al. (2020). Imple­men­ta­tion Mat­ters in Deep Policy Gra­di­ents: A Case Study on PPO and TRPO. arXiv:2005.12729.
660. Marcin Andrychow­icz, Anton Raichuk, Piotr Stańczyk et al. (2020). What Mat­ters In On-Policy Rein­force­ment Learn­ing? A Large-Scale Empir­i­cal Study. arXiv:2006.05990.
661. Jian Hu, Jason Klein Liu, Hao­tian Xu and Wei Shen (2025). REINFORCE++: Sta­bi­liz­ing Critic-Free Policy Opti­miza­tion with Global Advan­tage Nor­mal­iza­tion. arXiv:2501.03262.
662. Zichen Liu, Changyu Chen, Wenjun Li et al. (2025). Under­stand­ing R1-Zero-Like Train­ing: A Crit­i­cal Per­spec­tive. arXiv:2503.20783.
663. verl con­trib­u­tors (2025). verl/verl/trainer/ppo/core_algos.py at main · verl-project/verl. GitHub repos­i­tory source file.
664. Jujie He, Jiacai Liu, Chris Yuhao Liu et al. (2025). Sky­work Open Rea­soner 1 Tech­ni­cal Report. arXiv:2505.22312.
665. Wei Xiong, Jiarui Yao, Yuhui Xu et al. (2025). A Min­i­mal­ist Approach to LLM Rea­son­ing: from Rejec­tion Sam­pling to Rein­force. arXiv:2504.11343.
666. Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi and David Acuna (2026). MagicBeanStalk: Escap­ing the RLVR Zero Advan­tage Trap with Recur­sive Data Syn­the­sis. arXiv:2610.12659.
667. Zihe Liu, Jiashun Liu, Yancheng He et al. (2025). Part I: Tricks or Traps? A Deep Dive into RL for LLM Rea­son­ing. arXiv:2508.08221.
668. Mis­tral-AI, Abhi­nav Ras­togi, Albert Q. Jiang et al. (2025). Magis­tral. arXiv:2506.10910.
669. Max­well Nye, Anders Johan Andreassen, Guy Gur-Ari et al. (2021). Show Your Work: Scratch­pads for Inter­me­di­ate Com­pu­ta­tion with Lan­guage Models. arXiv:2112.00114.
670. Jason Wei, Xuezhi Wang, Dale Schu­ur­mans et al. (2022). Chain-of-Thought Prompt­ing Elic­its Rea­son­ing in Large Lan­guage Models. arXiv:2201.11903.
671. OpenAI, Aaron Jaech, Adam Kalai et al. (2024). OpenAI o1 System Card. arXiv:2412.16720.
672. Weihao Zeng, Yuzhen Huang, Qian Liu et al. (2025). SimpleRL-Zoo: Inves­ti­gat­ing and Taming Zero Rein­force­ment Learn­ing for Open Base Models in the Wild. arXiv:2503.18892.
673. Kan­ishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile and Noah D. Good­man (2025). Cog­ni­tive Behav­iors that Enable Self-Improv­ing Rea­son­ers, or, Four Habits of Highly Effec­tive STaRs. arXiv:2503.01307.
674. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig and Xiang Yue (2025). Demys­ti­fy­ing Long Chain-of-Thought Rea­son­ing in LLMs. arXiv:2502.03373.
675. Yiping Wang, Qing Yang, Zhiyuan Zeng et al. (2025). Rein­force­ment Learn­ing for Rea­son­ing in Large Lan­guage Models with One Train­ing Exam­ple. arXiv:2504.20571.
676. Etash Guha, Ryan Marten, Sedrick Keh et al. (2025). OpenThoughts: Data Recipes for Rea­son­ing Models. arXiv:2506.04178.
677. Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan and Lin Yan (2025). What's Behind PPO's Col­lapse in Long-CoT? Value Opti­miza­tion Holds the Secret. arXiv:2503.01491.
678. Chujie Zheng, Shix­uan Liu, Mingze Li et al. (2025). Group Sequence Policy Opti­miza­tion. arXiv:2507.18071.
679. Wenhan Ma, Hailin Zhang, Liang Zhao et al. (2025). Sta­bi­liz­ing MoE Rein­force­ment Learn­ing by Align­ing Train­ing and Infer­ence Routers. arXiv:2510.11370.
680. Ganqu Cui, Yuchen Zhang, Jiacheng Chen et al. (2025). The Entropy Mech­a­nism of Rein­force­ment Learn­ing for Rea­son­ing Lan­guage Models. arXiv:2505.22617.
681. DeepSeek-AI, Aixin Liu, Aoxue Mei et al. (2025). DeepSeek-V3.2: Push­ing the Fron­tier of Open Large Lan­guage Models. arXiv:2512.02556.
682. Shen­zhi Wang, Le Yu, Chang Gao et al. (2025). Beyond the 80/20 Rule: High-Entropy Minor­ity Tokens Drive Effec­tive Rein­force­ment Learn­ing for LLM Rea­son­ing. arXiv:2506.01939.
683. Penghui Qi, Zichen Liu, Xiangxin Zhou et al. (2025). Defeat­ing the Train­ing-Infer­ence Mis­match via FP16. arXiv:2510.26788.
684. Feng Yao and Liyuan Liu (2025). Flash-RL: Imple­men­ta­tion for FP8/INT8 Roll­out for RL train­ing with­out per­for­mence drop. GitHub repos­i­tory.
685. Lasse Espe­holt, Hubert Soyer, Remi Munos et al. (2018). IMPALA: Scal­able Dis­trib­uted Deep-RL with Impor­tance Weighted Actor-Learner Archi­tec­tures. arXiv:1802.01561.
686. Alexan­dre Piché, Ehsan Kamal­loo, Rafael Par­di­nas, Xiaoyin Chen and Dzmitry Bah­danau (2025). PipelineRL: Faster On-policy Rein­force­ment Learn­ing for Long Sequence Gen­er­a­tion. arXiv:2509.19128.
687. Yu Li, Yufa Zhou and Tian Lan (2026). On the Origin and Evo­lu­tion of Train­ing-Infer­ence Mis­match in LLM Rein­force­ment Learn­ing. arXiv:2610.40945.
688. Tian­run Yu, Kaix­i­ang Zhao, Shangzhe Li et al. (2026). Rethink­ing Train­ing-Infer­ence Mis­match in LLM Rein­force­ment Learn­ing: Where It Arises and How to Cor­rect It. arXiv:2609.32444.
689. Yax­i­ang Zhang, Yingru Li, Jiacai Liu et al. (2026). Beyond Pre­ci­sion: Train­ing-Infer­ence Mis­match is an Opti­miza­tion Prob­lem and Simple LR Sched­ul­ing Fixes It. arXiv:2602.01826.
690. Tai­heng Pan (2026). Cool the Sam­pler, Not the Learner: Sam­pling Tem­per­a­ture Moves the Stal­e­ness Cliff of Impor­tance-Cor­rected GRPO. arXiv:2609.36953.
691. Xuezhi Wang, Jason Wei, Dale Schu­ur­mans et al. (2022). Self-Con­sis­tency Improves Chain of Thought Rea­son­ing in Lan­guage Models. arXiv:2203.11171.
692. Xingyu Chen, Jiahao Xu, Tian Liang et al. (2024). Do NOT Think That Much for 2+3=? On the Over­think­ing of o1-Like LLMs. arXiv:2412.21187.
693. Wenjie Ma, Jingx­uan He, Char­lie Snell et al. (2025). Rea­son­ing Models Can Be Effec­tive With­out Think­ing. arXiv:2504.09858.
694. Ale­jan­dro Cuadron, Dacheng Li, Wenjie Ma et al. (2025). The Danger of Over­think­ing: Exam­in­ing the Rea­son­ing-Action Dilemma in Agen­tic Tasks. arXiv:2502.08235.
695. Yue Wang, Qiuzhi Liu, Jiahao Xu et al. (2025). Thoughts Are All Over the Place: On the Under­think­ing of o1-Like LLMs. arXiv:2501.18585.
696. Parshin Sho­jaee, Iman Mirzadeh, Keivan Alizadeh et al. (2025). The Illu­sion of Think­ing: Under­stand­ing the Strengths and Lim­i­ta­tions of Rea­son­ing Models via the Lens of Prob­lem Com­plex­ity. arXiv:2506.06941.
697. A. Lawsen (2025). Com­ment on The Illu­sion of Think­ing: Under­stand­ing the Strengths and Lim­i­ta­tions of Rea­son­ing Models via the Lens of Prob­lem Com­plex­ity. arXiv:2506.09250.
698. Amrith Setlur, Nived Rajara­man, Sergey Levine and Aviral Kumar (2025). Scal­ing Test-Time Com­pute With­out Ver­i­fi­ca­tion or RL is Sub­op­ti­mal. arXiv:2502.12118.
699. Mark Chen, Jerry Tworek, Heewoo Jun et al. (2021). Eval­u­at­ing Large Lan­guage Models Trained on Code. arXiv:2107.03374.
700. Xinhao Yao, Lu Yu, Xiaolin Hu et al. (2025). The Debate on RLVR Rea­son­ing Capa­bil­ity Bound­ary: Shrink­age, Expan­sion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028.
701. Fang Wu, Weihao Xuan, Danny Xing et al. (2025). The Invis­i­ble Leash: Why RLVR May or May Not Escape Its Origin. arXiv:2507.14843.
702. Xumeng Wen, Zihan Liu, Shun Zheng et al. (2025). Rein­force­ment Learn­ing with Ver­i­fi­able Rewards Implic­itly Incen­tivizes Cor­rect Rea­son­ing in Base LLMs. arXiv:2506.14245.
703. Char­lie Zhang, Graham Neubig and Xiang Yue (2025). On the Inter­play of Pre-Train­ing, Mid-Train­ing, and RL on Rea­son­ing Lan­guage Models. arXiv:2512.07783.
704. Mingqi Wu, Zhihao Zhang, Qiaole Dong et al. (2025). Rea­son­ing or Mem­o­riza­tion? Unre­li­able Results of Rein­force­ment Learn­ing Due to Data Con­tam­i­na­tion. arXiv:2507.10532.
705. Zhewei Yao, Reza Yaz­dani Aminabadi, Olatunji Ruwase et al. (2023). DeepSpeed-Chat: Easy, Fast and Afford­able RLHF Train­ing of ChatGPT-like Models at All Scales. arXiv:2308.01320.
706. Jian Hu, Xibin Wu, Wei Shen et al. (2024). OpenRLHF: An Easy-to-use, Scal­able and High-per­for­mance RLHF Frame­work. arXiv:2405.11143.
707. Guang­ming Sheng, Chi Zhang, Zil­ingfeng Ye et al. (2024). HybridFlow: A Flex­i­ble and Effi­cient RLHF Frame­work. arXiv:2409.19256.
708. Gerald Shen, Zhilin Wang, Olivier Delal­leau et al. (2024). NeMo-Aligner: Scal­able Tool­kit for Effi­cient Model Align­ment. arXiv:2405.01481.
709. Wei Fu, Jiax­uan Gao, Xujie Shen et al. (2025). AReaL: A Large-Scale Asyn­chro­nous Rein­force­ment Learn­ing System for Lan­guage Rea­son­ing. arXiv:2505.24298.
710. Yinmin Zhong, Zili Zhang, Xiao­niu Song et al. (2025). StreamRL: Scal­able, Het­ero­ge­neous, and Elas­tic RL for LLMs with Dis­ag­gre­gated Stream Gen­er­a­tion. arXiv:2504.15930.
711. verl con­trib­u­tors (2024). verl/HybridFlow: A Flex­i­ble and Effi­cient RL Post-Train­ing Frame­work. GitHub repos­i­tory.
712. Bo Wu, Sid Wang, Yunhao Tang et al. (2025). LlamaRL: A Dis­trib­uted Asyn­chro­nous Rein­force­ment Learn­ing Frame­work for Effi­cient Large-scale LLM Train­ing. arXiv:2505.24034.
713. Michael Noukhovitch, Shengyi Huang, Sophie Xhon­neux et al. (2024). Asyn­chro­nous RLHF: Faster and More Effi­cient Off-Policy RL for Lan­guage Models. arXiv:2410.18252.
714. Yuhang Xu, Kaibin Tian, Yang Tian et al. (2026). BubbleSpec: Turn­ing Long-Tail Bub­bles into Spec­u­la­tive Roll­out Drafts for Syn­chro­nous Rein­force­ment Learn­ing. arXiv:2605.08862.
715. OpenAI, Christo­pher Berner, Greg Brock­man et al. (2019). Dota 2 with Large Scale Deep Rein­force­ment Learn­ing. arXiv:1912.06680.
716. Lean­dro von Werra, Younes Belkada, Lewis Tun­stall et al. (2020). TRL: Trans­form­ers Rein­force­ment Learn­ing. GitHub repos­i­tory.
717. Weixun Wang, Shaopan Xiong, Gengru Chen et al. (2025). Rein­force­ment Learn­ing Opti­miza­tion for Large-Scale Learn­ing: An Effi­cient and User-Friendly Scal­ing Library. arXiv:2506.06122.
718. Shiyi Cao, Dacheng Li, Fangzhou Zhao et al. (2025). SkyRL-Agent: Effi­cient RL Train­ing for Multi-turn LLM Agent. arXiv:2511.16108.
719. Rei­ichiro Nakano, Jacob Hilton, Suchir Balaji et al. (2021). WebGPT: Browser-assisted ques­tion-answer­ing with human feed­back. arXiv:2112.09332.
720. Shunyu Yao, Jef­frey Zhao, Dian Yu et al. (2022). ReAct: Syn­er­giz­ing Rea­son­ing and Acting in Lan­guage Models. arXiv:2210.03629.
721. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì et al. (2023). Tool­former: Lan­guage Models Can Teach Them­selves to Use Tools. arXiv:2302.04761.
722. Jiazhan Feng, Shijue Huang, Xing­wei Qu et al. (2025). ReTool: Rein­force­ment Learn­ing for Strate­gic Tool Use in LLMs. arXiv:2504.11536.
723. Jiayi Pan, Xingyao Wang, Graham Neubig et al. (2024). Train­ing Soft­ware Engi­neer­ing Agents and Ver­i­fiers with SWE-Gym. arXiv:2412.21139.
724. Yux­i­ang Wei, Olivier Duchenne, Jade Copet et al. (2025). SWE-RL: Advanc­ing LLM Rea­son­ing via Rein­force­ment Learn­ing on Open Soft­ware Evo­lu­tion. arXiv:2502.18449.
725. Naman Jain, Jaski­rat Singh, Manish Shetty et al. (2025). R2E-Gym: Pro­ce­dural Envi­ron­ments and Hybrid Ver­i­fiers for Scal­ing Open-Weights SWE Agents. arXiv:2504.07164.
726. Bowen Jin, Hansi Zeng, Zhen­rui Yue et al. (2025). Search-R1: Train­ing LLMs to Reason and Lever­age Search Engines with Rein­force­ment Learn­ing. arXiv:2503.09516.
727. Lang Feng, Zheng­hai Xue, Tingcong Liu and Bo An (2025). Group-in-Group Policy Opti­miza­tion for LLM Agent Train­ing. arXiv:2505.10978.
728. Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine and Aviral Kumar (2024). ArCHer: Train­ing Lan­guage Model Agents via Hier­ar­chi­cal Multi-Turn RL. arXiv:2402.19446.
729. Zihan Wang, Kan­grui Wang, Qineng Wang et al. (2025). RAGEN: Under­stand­ing Self-Evo­lu­tion in LLM Agents via Multi-Turn Rein­force­ment Learn­ing. arXiv:2504.20073.
730. Vic­to­ria Krakovna, Jonathan Uesato, Vladimir Miku­lik et al. (2020). Spec­i­fi­ca­tion gaming: the flip side of AI inge­nu­ity. DeepMind blog.
731. Alexan­der Pan, Kush Bhatia and Jacob Stein­hardt (2022). The Effects of Reward Mis­spec­i­fi­ca­tion: Map­ping and Mit­i­gat­ing Mis­aligned Models. arXiv:2201.03544.
732. Sydney Von Arx, Lawrence Chan and Beth Barnes (2025). Recent Fron­tier Models Are Reward Hack­ing. METR blog.
733. Bowen Baker, Joost Huizinga, Leo Gao et al. (2025). Mon­i­tor­ing Rea­son­ing Models for Mis­be­hav­ior and the Risks of Pro­mot­ing Obfus­ca­tion. arXiv:2503.11926.
734. Tomek Korbak, Mikita Balesni, Eliz­a­beth Barnes et al. (2025). Chain of Thought Mon­i­tora­bil­ity: A New and Frag­ile Oppor­tu­nity for AI Safety. arXiv:2507.11473.
735. Tianyi Lorena Yan and John Hewitt (2026). Reward Mis­spec­i­fi­ca­tion Degrades Chain-of-Thought Mon­i­tora­bil­ity. arXiv:2610.28303.
736. Miles Turpin, Julian Michael, Ethan Perez and Samuel R. Bowman (2023). Lan­guage Models Don't Always Say What They Think: Unfaith­ful Expla­na­tions in Chain-of-Thought Prompt­ing. arXiv:2305.04388.
737. Tamera Lanham, Anna Chen, Ansh Radha­krish­nan et al. (2023). Mea­sur­ing Faith­ful­ness in Chain-of-Thought Rea­son­ing. arXiv:2307.13702.
738. Yanda Chen, Joe Benton, Ansh Radha­krish­nan et al. (2025). Rea­son­ing Models Don't Always Say What They Think. arXiv:2505.05410.
739. Melody Y. Guan, Miles Wang, Micah Car­roll et al. (2025). Mon­i­tor­ing Mon­i­tora­bil­ity. arXiv:2512.18311.
740. Anthropic (2025). System Card: Claude Opus 4 & Claude Sonnet 4. Anthropic system card.
741. Arun Jose and Julian Stastny (2026). Shal­low Beliefs: Syn­thetic doc­u­ment fine­tun­ing does not inoc­u­late against emer­gent mis­align­ment from reward hack­ing. arXiv:2609.14998.
742. Clémen­tine Four­rier, Nathan Habib, Julien Launay and Thomas Wolf (2023). What's going on with the Open LLM Leader­board?. Hug­ging Face blog.
743. Yuzhen Huang, Jing­han Zhang, Zifei Shan and Junx­ian He (2024). Com­pres­sion Rep­re­sents Intel­li­gence Lin­early. arXiv:2404.09937.
744. Rowan Zellers, Ari Holtz­man, Yonatan Bisk, Ali Farhadi and Yejin Choi (2019). HellaSwag: Can a Machine Really Finish Your Sen­tence?. arXiv:1905.07830.
745. Lovish Madaan, Aaditya K. Singh, Rylan Scha­ef­fer et al. (2024). Quan­ti­fy­ing Vari­ance in Eval­u­a­tion Bench­marks. arXiv:2406.10229.
746. Keisuke Sak­aguchi, Ronan Le Bras, Chan­dra Bha­ga­vat­ula and Yejin Choi (2019). WinoGrande: An Adver­sar­ial Wino­grad Schema Chal­lenge at Scale. arXiv:1907.10641.
747. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jian­feng Gao and Yejin Choi (2019). PIQA: Rea­son­ing about Phys­i­cal Com­mon­sense in Nat­u­ral Lan­guage. arXiv:1911.11641.
748. Mandar Joshi, Eunsol Choi, Daniel S. Weld and Luke Zettle­moyer (2017). TriviaQA: A Large Scale Dis­tantly Super­vised Chal­lenge Dataset for Read­ing Com­pre­hen­sion. arXiv:1705.03551.
749. Peter Clark, Isaac Cowhey, Oren Etzioni et al. (2018). Think you have Solved Ques­tion Answer­ing? Try ARC, the AI2 Rea­son­ing Chal­lenge. arXiv:1803.05457.
750. Mirac Suzgun, Nathan Scales, Nathanael Schärli et al. (2022). Chal­leng­ing BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. arXiv:2210.09261.
751. Dan Hendrycks, Collin Burns, Saurav Kada­vath et al. (2021). Mea­sur­ing Math­e­mat­i­cal Prob­lem Solv­ing With the MATH Dataset. arXiv:2103.03874.
752. Jacob Austin, Augus­tus Odena, Max­well Nye et al. (2021). Pro­gram Syn­the­sis with Large Lan­guage Models. arXiv:2108.07732.
753. David Rein, Betty Li Hou, Asa Cooper Stick­land et al. (2023). GPQA: A Grad­u­ate-Level Google-Proof Q&A Bench­mark. arXiv:2311.12022.
754. Mubashara Akhtar, Anka Reuel, Prajna Soni et al. (2026). When AI Bench­marks Plateau: A Sys­tem­atic Study of Bench­mark Sat­u­ra­tion. arXiv:2602.16763.
755. Wikipedia con­trib­u­tors (2026). Amer­i­can Invi­ta­tional Math­e­mat­ics Exam­i­na­tion. Wikipedia.
756. Elliot Glazer, Ege Erdil, Tamay Besiroglu et al. (2024). FrontierMath: A Bench­mark for Eval­u­at­ing Advanced Math­e­mat­i­cal Rea­son­ing in AI. arXiv:2411.04872.
757. Long Phan, Alice Gatti, Ziwen Han et al. (2025). Human­ity's Last Exam. arXiv:2501.14249.
758. Fran­cois Chol­let, Mike Knoop, Gre­gory Kam­radt and Bryan Lan­ders (2024). ARC Prize 2024: Tech­ni­cal Report. arXiv:2412.04604.
759. François Chol­let (2024). OpenAI o3 Break­through High Score on ARC-AGI-Pub. ARC Prize blog.
760. Fran­cois Chol­let, Mike Knoop, Gre­gory Kam­radt, Bryan Lan­ders and Henry Pinkard (2025). ARC-AGI-2: A New Chal­lenge for Fron­tier AI Rea­son­ing Sys­tems. arXiv:2505.11831.
761. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi et al. (2024). GSM-Sym­bolic: Under­stand­ing the Lim­i­ta­tions of Math­e­mat­i­cal Rea­son­ing in Large Lan­guage Models. arXiv:2410.05229.
762. Jiawei Liu, Chun­qiu Steven Xia, Yuyao Wang and Ling­ming Zhang (2023). Is Your Code Gen­er­ated by ChatGPT Really Cor­rect? Rig­or­ous Eval­u­a­tion of Large Lan­guage Models for Code Gen­er­a­tion. arXiv:2305.01210.
763. Simon Ott, Adri­ano Bar­bosa-Silva, Kathrin Blagec, Jan Brauner and Matthias Samwald (2022). Map­ping global dynam­ics of bench­mark cre­ation and sat­u­ra­tion in arti­fi­cial intel­li­gence. arXiv:2203.04592.
764. Mehran Kazemi, Bahare Fatemi, Hritik Bansal et al. (2025). BIG-Bench Extra Hard. arXiv:2502.19187.
765. Colin White, Samuel Dooley, Manley Roberts et al. (2024). LiveBench: A Chal­leng­ing, Con­tam­i­na­tion-Lim­ited LLM Bench­mark. arXiv:2406.19314.
766. Mislav Balunović, Jasper Dekon­inck, Ivo Petrov, Nikola Jovanović and Martin Vechev (2025). MathArena: Eval­u­at­ing LLMs on Uncon­tam­i­nated Math Com­pe­ti­tions. arXiv:2505.23281.
767. Michael Skar­lin­ski, Jon Lau­rent, Albert Bou and Andrew White (2025). About 30% of Human­ity's Last Exam chem­istry/biol­ogy answers are likely wrong. FutureHouse blog.
768. Weiqi Zhai, Zhihai Wang, Jing­hang Wang et al. (2026). HLE-Ver­i­fied: A Sys­tem­atic Ver­i­fi­ca­tion and Struc­tured Revi­sion of Human­ity's Last Exam. arXiv:2602.13964.
769. Melanie Sclar, Yejin Choi, Yulia Tsvetkov and Alane Suhr (2023). Quan­ti­fy­ing Lan­guage Models' Sen­si­tiv­ity to Spu­ri­ous Fea­tures in Prompt Design or: How I learned to start wor­ry­ing about prompt for­mat­ting. arXiv:2310.11324.
770. Chujie Zheng, Hao Zhou, Fan­dong Meng, Jie Zhou and Minlie Huang (2023). Large Lan­guage Models Are Not Robust Mul­ti­ple Choice Selec­tors. arXiv:2309.03882.
771. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong et al. (2024). Are We Done with MMLU?. arXiv:2406.04127.
772. Deep Gan­guli, Nicholas Schiefer, Marina Favaro and Jack Clark (2023). Chal­lenges in eval­u­at­ing AI sys­tems. Anthropic blog.
773. Hugh Zhang, Jeff Da, Dean Lee et al. (2024). A Care­ful Exam­i­na­tion of Large Lan­guage Model Per­for­mance on Grade School Arith­metic. arXiv:2405.00332.
774. Berk Atil, Sarp Aykent, Alexa Chit­tams et al. (2024). Non-Deter­min­ism of "Deter­min­is­tic" LLM Set­tings. arXiv:2408.04667.
775. Stella Bider­man, Hailey Schoelkopf, Lin­tang Sutawika et al. (2024). Lessons from the Trenches on Repro­ducible Eval­u­a­tion of Lan­guage Models. arXiv:2405.14782.
776. Andreas Hochlehn­ert, Hardik Bhat­na­gar, Vishaal Udan­darao et al. (2025). A Sober Look at Progress in Lan­guage Model Rea­son­ing: Pit­falls and Paths to Repro­ducibil­ity. arXiv:2504.07086.
777. Evan Miller (2024). Adding Error Bars to Evals: A Sta­tis­ti­cal Approach to Lan­guage Model Eval­u­a­tions. arXiv:2411.00640.
778. Pranav Singh and Prashant Singh (2026). What Fixed-Roll­out pass@k Eval­u­a­tions Can Iden­tify. arXiv:2609.09245.
779. Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan (2024). τ-bench: A Bench­mark for Tool-Agent-User Inter­ac­tion in Real-World Domains. arXiv:2406.12045.
780. METR (2026). Task-Com­ple­tion Time Hori­zons of Fron­tier AI Models. METR.
781. Sam Bowyer, Lau­rence Aitchi­son and Desi R. Ivanova (2025). Posi­tion: Don't Use the CLT in LLM Evals With Fewer Than a Few Hun­dred Dat­a­points. arXiv:2503.01747.
782. Dallas Card, Peter Hen­der­son, Urvashi Khan­del­wal et al. (2020). With Little Power Comes Great Respon­si­bil­ity. arXiv:2010.06595.
783. Cheng Xu, Shuhao Guan, Derek Greene and M-Tahar Kechadi (2024). Bench­mark Data Con­tam­i­na­tion of Large Lan­guage Models: A Survey. arXiv:2406.04244.
784. Inbal Magar and Roy Schwartz (2022). Data Con­tam­i­na­tion: From Mem­o­riza­tion to Exploita­tion. arXiv:2203.08242.
785. Simone Bal­loccu, Patrícia Schmid­tová, Mateusz Lango and Ondřej Dušek (2024). Leak, Cheat, Repeat: Data Con­tam­i­na­tion and Eval­u­a­tion Mal­prac­tices in Closed-Source LLMs. arXiv:2402.03927.
786. Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poul­ton et al. (2024). Eval­u­a­tion data con­tam­i­na­tion in LLMs: how do we mea­sure it and (when) does it matter?. arXiv:2411.03923.
787. Weijia Shi, Anirudh Ajith, Mengzhou Xia et al. (2023). Detect­ing Pre­train­ing Data from Large Lan­guage Models. arXiv:2310.16789.
788. Jingyang Zhang, Jing­wei Sun, Eric Yeats et al. (2024). Min-K%++: Improved Base­line for Detect­ing Pre-Train­ing Data from Large Lan­guage Models. arXiv:2404.02936.
789. Michael Duan, Anshu­man Suri, Niloo­far Mireshghal­lah et al. (2024). Do Mem­ber­ship Infer­ence Attacks Work on Large Lan­guage Models?. arXiv:2402.07841.
790. Yonatan Oren, Nicole Meis­ter, Niladri Chat­terji, Faisal Ladhak and Tat­sunori B. Hashimoto (2023). Prov­ing Test Set Con­tam­i­na­tion in Black Box Lan­guage Models. arXiv:2310.17623.
791. Aarohi Sri­vas­tava, Abhi­nav Ras­togi, Abhishek Rao et al. (2022). Beyond the Imi­ta­tion Game: Quan­ti­fy­ing and extrap­o­lat­ing the capa­bil­i­ties of lan­guage models. arXiv:2206.04615.
792. Alon Jacovi, Avi Caci­u­laru, Omer Gold­man and Yoav Gold­berg (2023). Stop Upload­ing Test Data in Plain Text: Prac­ti­cal Strate­gies for Mit­i­gat­ing Data Con­tam­i­na­tion by Eval­u­a­tion Bench­marks. arXiv:2305.10160.
793. Yongjie Wang, Xinyue Zhang, Kun­hong Yao et al. (2026). Search-Time Con­tam­i­na­tion in Deep Research Agents: Mea­sur­ing Per­for­mance Infla­tion in Public Bench­mark Eval­u­a­tion. arXiv:2606.05241.
794. Lian­min Zheng, Wei-Lin Chiang, Ying Sheng et al. (2023). Judg­ing LLM-as-a-Judge with MT-Bench and Chat­bot Arena. arXiv:2306.05685.
795. Aman Singh Thakur, Kartik Choud­hary, Venkat Srinik Ramaya­pally, Sankaran Vaidyanathan and Dieuwke Hupkes (2024). Judg­ing the Judges: Eval­u­at­ing Align­ment and Vul­ner­a­bil­i­ties in LLMs-as-Judges. arXiv:2406.12624.
796. Sijun Tan, Siyuan Zhuang, Kyle Mont­gomery et al. (2024). JudgeBench: A Bench­mark for Eval­u­at­ing LLM-based Judges. arXiv:2410.12784.
797. Arjun Pan­ickssery, Samuel R. Bowman and Shi Feng (2024). LLM Eval­u­a­tors Rec­og­nize and Favor Their Own Gen­er­a­tions. arXiv:2404.13076.
798. Peiyi Wang, Lei Li, Liang Chen et al. (2023). Large Lan­guage Models are not Fair Eval­u­a­tors. arXiv:2305.17926.
799. Koki Wataoka, Tsub­asa Taka­hashi and Ryokan Ri (2024). Self-Pref­er­ence Bias in LLM-as-a-Judge. arXiv:2410.21819.
800. Dawei Li, Ren­liang Sun, Yue Huang et al. (2025). Pref­er­ence Leak­age: A Con­tam­i­na­tion Prob­lem in LLM-as-a-judge. arXiv:2502.01534.
801. Xiaosen Zheng, Tianyu Pang, Chao Du et al. (2024). Cheat­ing Auto­matic LLM Bench­marks: Null Models Achieve High Win Rates. arXiv:2410.07137.
802. Anas­ta­sios N. Angelopou­los, Stephen Bates, Clara Fan­njiang, Michael I. Jordan and Tijana Zrnic (2023). Pre­dic­tion-Pow­ered Infer­ence. arXiv:2301.09633.
803. Wei-Lin Chiang, Lian­min Zheng, Ying Sheng et al. (2024). Chat­bot Arena: An Open Plat­form for Eval­u­at­ing LLMs by Human Pref­er­ence. arXiv:2403.04132.
804. Meriem Boub­dir, Edward Kim, Beyza Ermis, Sara Hooker and Marzieh Fadaee (2023). Elo Uncov­ered: Robust­ness and Best Prac­tices in Lan­guage Model Eval­u­a­tion. arXiv:2311.17295.
805. Shiv­a­lika Singh, Yiyang Nan, Alex Wang et al. (2025). The Leader­board Illu­sion. arXiv:2504.20879.
806. Tianle Li, Anas­ta­sios Angelopou­los and Wei-Lin Chiang (2024). Does style matter? Dis­en­tan­gling style and sub­stance in Chat­bot Arena. LMSYS blog.
807. Tianle Li, Wei-Lin Chiang, Evan Frick et al. (2024). From Crowd­sourced Data to High-Qual­ity Bench­marks: Arena-Hard and BenchBuilder Pipe­line. arXiv:2406.11939.
808. Bill Yuchen Lin, Yun­tian Deng, Khy­athi Chandu et al. (2024). WildBench: Bench­mark­ing LLMs with Chal­leng­ing Tasks from Real Users in the Wild. arXiv:2406.04770.
809. Jef­frey Zhou, Tian­jian Lu, Swa­roop Mishra et al. (2023). Instruc­tion-Fol­low­ing Eval­u­a­tion for Large Lan­guage Models. arXiv:2311.07911.
810. Valentina Pyatkin, Saumya Malik, Vic­to­ria Graf et al. (2025). Gen­er­al­iz­ing Ver­i­fi­able Instruc­tion Fol­low­ing. arXiv:2507.02833.
811. Yangsibo Huang, Milad Nasr, Anas­ta­sios Angelopou­los et al. (2025). Explor­ing and Mit­i­gat­ing Adver­sar­ial Manip­u­la­tion of Voting-Based Leader­boards. arXiv:2501.07493.
812. Soovadeep Bakshi, Apurba Bose, Aish­warya Bose and Sandip Nandi (2026). Sim­i­lar Models, Broken Odds: Where Luce's Axiom Fails in Chat­bot Arena. arXiv:2610.19095.
813. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan (2024). AI Agents That Matter. arXiv:2407.01502.
814. Carlos E. Jimenez, John Yang, Alexan­der Wettig et al. (2023). SWE-bench: Can Lan­guage Models Resolve Real-World GitHub Issues?. arXiv:2310.06770.
815. Mike A. Mer­rill, Alexan­der G. Shaw, Nicholas Car­lini et al. (2026). Ter­mi­nal-Bench: Bench­mark­ing Agents on Hard, Real­is­tic Tasks in Com­mand Line Inter­faces. arXiv:2601.11868.
816. UK AI Secu­rity Insti­tute (2024). Inspect. Inspect doc­u­men­ta­tion.
817. Prince­ton NLP (2024). prince­ton-nlp/SWE-bench_Ver­i­fied. Hug­ging Face dataset card.
818. Yuxuan Zhu, Tengjun Jin, Yada Pruk­sachatkun et al. (2025). Estab­lish­ing Best Prac­tices for Build­ing Rig­or­ous Agen­tic Bench­marks. arXiv:2507.02825.
819. Chun­qiu Steven Xia, Yinlin Deng, Soren Dunn and Ling­ming Zhang (2024). Agent­less: Demys­ti­fy­ing LLM-based Soft­ware Engi­neer­ing Agents. arXiv:2407.01489.
820. Xiang Deng, Jeff Da, Edwin Pan et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Hori­zon Soft­ware Engi­neer­ing Tasks?. arXiv:2509.16941.
821. Tian­bao Xie, Danyang Zhang, Jixuan Chen et al. (2024). OSWorld: Bench­mark­ing Mul­ti­modal Agents for Open-Ended Tasks in Real Com­puter Envi­ron­ments. arXiv:2404.07972.
822. Shuyan Zhou, Frank F. Xu, Hao Zhu et al. (2023). WebArena: A Real­is­tic Web Envi­ron­ment for Build­ing Auton­o­mous Agents. arXiv:2307.13854.
823. Grégoire Mialon, Clémen­tine Four­rier, Craig Swift et al. (2023). GAIA: a bench­mark for Gen­eral AI Assis­tants. arXiv:2311.12983.
824. Jun Shern Chan, Neil Chowd­hury, Oliver Jaffe et al. (2024). MLE-bench: Eval­u­at­ing Machine Learn­ing Agents on Machine Learn­ing Engi­neer­ing. arXiv:2410.07095.
825. Hjal­mar Wijk, Tao Lin, Joel Becker et al. (2024). RE-Bench: Eval­u­at­ing fron­tier AI R&D capa­bil­i­ties of lan­guage model agents against human experts. arXiv:2411.15114.
826. Giulio Starace, Oliver Jaffe, Dane Sher­burn et al. (2025). PaperBench: Eval­u­at­ing AI's Abil­ity to Repli­cate AI Research. arXiv:2504.01848.
827. David Rein, Joel Becker, Amy Deng et al. (2025). HCAST: Human-Cal­i­brated Auton­omy Soft­ware Tasks. arXiv:2503.17354.
828. Tom David­son, Jean-Stanis­las Denain, Pablo Vil­lalo­bos and Guillem Bas (2023). AI capa­bil­i­ties can be sig­nif­i­cantly improved with­out expen­sive retrain­ing. arXiv:2312.07413.
829. Thomas Kwa and Vin­cent Cheng (2025). How Does Time Hori­zon Vary Across Domains?. METR blog.
830. Joel Becker, Nate Rush, Eliz­a­beth Barnes and David Rein (2025). Mea­sur­ing the Impact of Early-2025 AI on Expe­ri­enced Open-Source Devel­oper Pro­duc­tiv­ity. arXiv:2507.09089.
831. Toby Shevlane, Sebas­tian Far­quhar, Ben Garfinkel et al. (2023). Model eval­u­a­tion for extreme risks. arXiv:2305.15324.
832. Mary Phuong, Matthew Aitchi­son, Elliot Catt et al. (2024). Eval­u­at­ing Fron­tier Models for Dan­ger­ous Capa­bil­i­ties. arXiv:2403.13793.
833. Megan Kin­ni­ment, Lucas Jun Koba Sato, Haox­ing Du et al. (2023). Eval­u­at­ing Lan­guage-Model Agents on Real­is­tic Auton­o­mous Tasks. arXiv:2312.11671.
834. Nathaniel Li, Alexan­der Pan, Anjali Gopal et al. (2024). The WMDP Bench­mark: Mea­sur­ing and Reduc­ing Mali­cious Use With Unlearn­ing. arXiv:2403.03218.
835. Andy K. Zhang, Neil Perry, Riya Dulepet et al. (2024). Cybench: A Frame­work for Eval­u­at­ing Cyber­se­cu­rity Capa­bil­i­ties and Risks of Lan­guage Models. arXiv:2408.08926.
836. OpenAI (2025). Pre­pared­ness Frame­work, Ver­sion 2. OpenAI policy doc­u­ment.
837. Anthropic (2023). Anthropic's Respon­si­ble Scal­ing Policy, Ver­sion 1.0. Anthropic policy doc­u­ment.
838. Anthropic (2024). Our updated Respon­si­ble Scal­ing Policy. Anthropic news.
839. Anthropic (2026). Respon­si­ble Scal­ing Policy Ver­sion 3.0. Anthropic policy doc­u­ment.
840. Anthropic (2026). Anthropic's Respon­si­ble Scal­ing Policy: Ver­sion 3.0. Anthropic news.
841. Google DeepMind (2025). Updat­ing the Fron­tier Safety Frame­work. Google DeepMind blog.
842. Four Flynn, Helen King and Anca Dragan (2025). Strength­en­ing our Fron­tier Safety Frame­work. Google DeepMind blog.
843. Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown and Fran­cis Rhys Ward (2024). AI Sand­bag­ging: Lan­guage Models can Strate­gi­cally Under­per­form on Eval­u­a­tions. arXiv:2406.07358.
844. Ryan Green­blatt, Fabien Roger, Dmitrii Krashenin­nikov and David Krueger (2024). Stress-Test­ing Capa­bil­ity Elic­i­ta­tion With Pass­word-Locked Models. arXiv:2405.19550.
845. Anthropic (2026). Anthropic's Respon­si­ble Scal­ing Policy. Anthropic policy page.
846. David Man­heim and Scott Garrabrant (2018). Cat­e­go­riz­ing Vari­ants of Good­hart's Law. arXiv:1803.04585.
847. PyTorch Con­trib­u­tors (2026). Dis­trib­uted Data Par­al­lel. PyTorch 2.15 doc­u­men­ta­tion (notes).
848. Horace He and Think­ing Machines Lab (2025). Defeat­ing Non­de­ter­min­ism in LLM Infer­ence. Con­nec­tion­ism (Think­ing Machines Lab blog).
849. Rajeev Thakur, Rolf Raben­seifner and William Gropp (2005). Opti­miza­tion of Col­lec­tive Com­mu­ni­ca­tion Oper­a­tions in MPICH. The Inter­na­tional Jour­nal of High Per­for­mance Com­put­ing Appli­ca­tions.
850. Andrew Gib­ian­sky (2017). Bring­ing HPC Tech­niques to Deep Learn­ing. Blog post (Baidu Research).
851. Pitch Patara­suk and Xin Yuan (2009). Band­width opti­mal all-reduce algo­rithms for clus­ters of work­sta­tions. Jour­nal of Par­al­lel and Dis­trib­uted Com­put­ing.
852. Syl­vain Jeaugey (2019). Mas­sively Scale Your Deep Learn­ing Train­ing with NCCL 2.4. NVIDIA Tech­ni­cal Blog.
853. Peter Sanders, Jochen Speck and Jesper Lars­son Träff (2009). Two-tree algo­rithms for full band­width broad­cast, reduc­tion and scan. Par­al­lel Com­put­ing.
854. Zhiyi Hu, Siyuan Shen, Tom­maso Bonato et al. (2025). Demys­ti­fy­ing NCCL: An In-depth Anal­y­sis of GPU Com­mu­ni­ca­tion Pro­to­cols and Algo­rithms. arXiv:2507.04786.
855. Shen Li, Yanli Zhao, Rohan Varma et al. (2020). PyTorch Dis­trib­uted: Expe­ri­ences on Accel­er­at­ing Data Par­al­lel Train­ing. arXiv:2006.15704.
856. Jacob Austin, Sholto Dou­glas, Roy Frostig et al. (2025). How to Par­al­lelize a Trans­former for Train­ing. How To Scale Your Model (online book).
857. NVIDIA (2026). Dis­trib­uted Opti­mizer. Mega­tron-LM doc­u­men­ta­tion (GitHub).
858. Guan­hua Wang, Heyang Qin, Sam Ade Jacobs et al. (2023). ZeRO++: Extremely Effi­cient Col­lec­tive Com­mu­ni­ca­tion for Giant Model Train­ing. arXiv:2306.10209.
859. Yanli Zhao, Andrew Gu, Rohan Varma et al. (2023). PyTorch FSDP: Expe­ri­ences on Scal­ing Fully Sharded Data Par­al­lel. arXiv:2304.11277.
860. PyTorch Con­trib­u­tors (2026). torch.dis­trib­uted.fsdp.fully_shard. PyTorch 2.15 doc­u­men­ta­tion.
861. Yifu Wang, Horace He, Less Wright et al. (2024). [Dis­trib­uted w/ TorchTitan] Intro­duc­ing Async Tensor Par­al­lel­ism in PyTorch. PyTorch Forums.
862. Shibo Wang, Jin­liang Wei, Amit Sabne et al. (2022). Over­lap Com­mu­ni­ca­tion with Depen­dent Com­pu­ta­tion via Decom­po­si­tion in Large Deep Learn­ing Models. Pro­ceed­ings of the 28th ACM Inter­na­tional Con­fer­ence on Archi­tec­tural Sup­port for Pro­gram­ming Lan­guages and Oper­at­ing Sys­tems, Volume 1.
863. Yan­ping Huang, You­long Cheng, Ankur Bapna et al. (2018). GPipe: Effi­cient Train­ing of Giant Neural Net­works using Pipe­line Par­al­lel­ism. arXiv:1811.06965.
864. Aaron Harlap, Deepak Narayanan, Amar Phan­ishayee et al. (2018). PipeDream: Fast and Effi­cient Pipe­line Par­al­lel DNN Train­ing. arXiv:1806.03377.
865. Deepak Narayanan, Amar Phan­ishayee, Kaiyu Shi, Xie Chen and Matei Zaharia (2020). Memory-Effi­cient Pipe­line-Par­al­lel DNN Train­ing. arXiv:2006.09503.
866. Penghui Qi, Xinyi Wan, Guangx­ing Huang and Min Lin (2023). Zero Bubble Pipe­line Par­al­lel­ism. arXiv:2401.10241.
867. Hao Liu, Matei Zaharia and Pieter Abbeel (2023). Ring Atten­tion with Block­wise Trans­form­ers for Near-Infi­nite Con­text. arXiv:2310.01889.
868. Hao Liu and Pieter Abbeel (2023). Block­wise Par­al­lel Trans­former for Large Con­text Models. arXiv:2305.19370.
869. William Bran­don, Anirud­dha Nrusimha, Kevin Qian et al. (2023). Striped Atten­tion: Faster Ring Atten­tion for Causal Trans­form­ers. arXiv:2311.09431.
870. Sam Ade Jacobs, Masahiro Tanaka, Cheng­ming Zhang et al. (2023). DeepSpeed Ulysses: System Opti­miza­tions for Enabling Train­ing of Extreme Long Sequence Trans­former Models. arXiv:2309.14509.
871. Jiarui Fang and Shangchun Zhao (2024). USP: A Uni­fied Sequence Par­al­lel­ism Approach for Long Con­text Gen­er­a­tive AI. arXiv:2405.07719.
872. Hao Ge, Junda Feng, Qi Huang et al. (2025). ByteScale: Effi­cient Scal­ing of LLM Train­ing with a 2048K Con­text Length on More Than 12,000 GPUs. arXiv:2502.21231.
873. Amy Yang, Jingyi Yang, Aya Ibrahim et al. (2024). Con­text Par­al­lel­ism for Scal­able Mil­lion-Token Infer­ence. arXiv:2411.01783.
874. Samyam Rajb­han­dari, Con­g­long Li, Zhewei Yao et al. (2022). DeepSpeed-MoE: Advanc­ing Mix­ture-of-Experts Infer­ence and Train­ing to Power Next-Gen­er­a­tion AI Scale. arXiv:2201.05596.
875. Changho Hwang, Wei Cui, Yifan Xiong et al. (2022). Tutel: Adap­tive Mix­ture-of-Experts at Scale. arXiv:2206.03382.
876. Cheng­gang Zhao, Shangyan Zhou, Liyue Zhang et al. (2025). DeepEP: an effi­cient expert-par­al­lel com­mu­ni­ca­tion library. GitHub repos­i­tory.
877. Sami Jaghouar, Jack Min Ong, Man­veer Basra et al. (2024). INTELLECT-1 Tech­ni­cal Report. arXiv:2412.01152.
878. Arthur Douil­lard, Yanislav Donchev, Keith Rush et al. (2025). Stream­ing DiLoCo with over­lap­ping com­mu­ni­ca­tion: Towards a Dis­trib­uted Free Lunch. arXiv:2501.18512.
879. H. Bren­dan McMahan, Eider Moore, Daniel Ramage, Seth Hamp­son and Blaise Agüera y Arcas (2016). Com­mu­ni­ca­tion-Effi­cient Learn­ing of Deep Net­works from Decen­tral­ized Data. arXiv:1602.05629.
880. Sebas­tian U. Stich (2018). Local SGD Con­verges Fast and Com­mu­ni­cates Little. arXiv:1805.09767.
881. Tao Lin, Sebas­tian U. Stich, Kumar Kshi­tij Patel and Martin Jaggi (2018). Don't Use Large Mini-Batches, Use Local SGD. arXiv:1808.07217.
882. Arthur Douil­lard, Qixuan Feng, Andrei A. Rusu et al. (2023). DiLoCo: Dis­trib­uted Low-Com­mu­ni­ca­tion Train­ing of Lan­guage Models. arXiv:2311.08105.
883. Zachary Charles, Gabriel Teston, Lucio Dery et al. (2025). Com­mu­ni­ca­tion-Effi­cient Lan­guage Model Train­ing Scales Reli­ably and Robustly: Scal­ing Laws for DiLoCo. arXiv:2503.09799.
884. Stefan Horoi, Ben­jamin Thérien, Guy Wolf and Eugene Belilovsky (2026). Can Model Merg­ing Improve Aggre­ga­tion in DiLoCo?. arXiv:2607.03011.
885. Bowen Peng, Lizhang Chen, Baiyu Su et al. (2024). DeMo: Decou­pled Momen­tum Opti­miza­tion. arXiv:2411.19870.
886. Hanlin Tang, Shao­duo Gan, Ammar Ahmad Awan et al. (2021). 1-bit Adam: Com­mu­ni­ca­tion Effi­cient Large-Scale Train­ing with Adam's Con­ver­gence Speed. arXiv:2102.02888.
887. Amir Sarfi, Ben­jamin Thérien, Joel Lidin and Eugene Belilovsky (2025). Over­com­ing the Com­mu­ni­ca­tion-Per­for­mance Trade­off in LLM Pre­train­ing. arXiv:2508.15706.
888. Jari Kolehmainen, Niko­lay Bla­goev, Semih Kara et al. (2025). NoLoCo: No-all-reduce Low Com­mu­ni­ca­tion Train­ing Method for Large Models. arXiv:2506.10911.
889. Yuanzhong Xu, HyoukJoong Lee, Dehao Chen et al. (2021). GSPMD: Gen­eral and Scal­able Par­al­leliza­tion for ML Com­pu­ta­tion Graphs. arXiv:2105.04663.
890. Lian­min Zheng, Zhuo­han Li, Hao Zhang et al. (2022). Alpa: Automat­ing Inter- and Intra-Oper­a­tor Par­al­lel­ism for Dis­trib­uted Deep Learn­ing. arXiv:2201.12023.
891. Zhihao Jia, Matei Zaharia and Alex Aiken (2018). Beyond Data and Model Par­al­lel­ism for Deep Neural Net­works. arXiv:1807.05358.
892. JAX authors (2026). Manual par­al­lel­ism with shard_map. JAX doc­u­men­ta­tion.
893. PyTorch team (2026). torchti­tan: A PyTorch native plat­form for train­ing gen­er­a­tive AI models. GitHub repos­i­tory.
894. NVIDIA (2026). Mega­tron-LM: Ongo­ing research train­ing trans­former models at scale. GitHub repos­i­tory.
895. NVIDIA (2026). Mega­tron-Bridge: Train­ing library for Mega­tron-based models with bidi­rec­tional Hug­ging Face con­ver­sion capa­bil­ity. GitHub repos­i­tory.
896. NVIDIA (2026). NVIDIA-NeMo/Speech: A scal­able gen­er­a­tive AI frame­work built for researchers and devel­op­ers. GitHub repos­i­tory.
897. DeepSpeed team (2026). DeepSpeed is a deep learn­ing opti­miza­tion library that makes dis­trib­uted train­ing and infer­ence easy, effi­cient, and effec­tive. GitHub repos­i­tory.
898. Google AI Hyper­com­puter (2026). max­text: A simple, per­for­mant, and scal­able Jax LLM. GitHub repos­i­tory.
899. Hug­ging Face (2026). nan­otron: Min­i­mal­is­tic large lan­guage model 3D-par­al­lel­ism train­ing. GitHub repos­i­tory.
900. Stan­ford CRFM and Marin com­mu­nity (2026). lev­an­ter: Leg­i­ble, Scal­able, Repro­ducible Foun­da­tion Models with Named Ten­sors and Jax. GitHub repos­i­tory.
901. Meta AI (2022). OPT-175B Log­book. metaseq GitHub repos­i­tory.
902. Gemini Team, Rohan Anil, Sebas­tian Borgeaud et al. (2023). Gemini: A Family of Highly Capa­ble Mul­ti­modal Models. arXiv:2312.11805.
903. Apos­to­los Koko­lis, Michael Kuch­nik, John Hoff­man et al. (2024). Revis­it­ing Reli­a­bil­ity in Large-Scale Machine Learn­ing Research Clus­ters. arXiv:2410.21680.
904. Borui Wan, Gao­hong Liu, Zuquan Song et al. (2025). Robust LLM Train­ing Infra­struc­ture at ByteDance. arXiv:2509.16293.
905. Qing­hao Hu, Zhisheng Ye, Zerui Wang et al. (2024). Char­ac­ter­i­za­tion of Large Lan­guage Model Devel­op­ment in the Dat­a­cen­ter. arXiv:2403.07648.
906. Bianca Schroeder and Garth A. Gibson (2010). A Large-Scale Study of Fail­ures in High-Per­for­mance Com­put­ing Sys­tems. IEEE Trans­ac­tions on Depend­able and Secure Com­put­ing.
907. Devesh Tiwari, Saurabh Gupta, James Rogers et al. (2015). Under­stand­ing GPU errors on large-scale HPC sys­tems and the impli­ca­tions for system design and oper­a­tion. 2015 IEEE 21st Inter­na­tional Sym­po­sium on High Per­for­mance Com­puter Archi­tec­ture (HPCA).
908. Yi He, Mike Hutton, Steven Chan et al. (2023). Under­stand­ing and Mit­i­gat­ing Hard­ware Fail­ures in Deep Learn­ing Train­ing Sys­tems. Pro­ceed­ings of the 50th Annual Inter­na­tional Sym­po­sium on Com­puter Archi­tec­ture.
909. Shengkun Cui, Archit Patke, Hung Nguyen et al. (2025). Story of Two GPUs: Char­ac­ter­iz­ing the Resilience of Hopper H100 and Ampere A100 GPUs. arXiv:2503.11901.
910. Harish Dat­ta­traya Dixit, Sneha Pend­harkar, Matt Beadon et al. (2021). Silent Data Cor­rup­tions at Scale. arXiv:2102.11245.
911. Peter H. Hochschild, Paul Turner, Jef­frey C. Mogul et al. (2021). Cores that don't count. Pro­ceed­ings of the Work­shop on Hot Topics in Oper­at­ing Sys­tems.
912. Sub­ha­sish Mitra, Subho Baner­jee, Martin Dixon et al. (2025). Silent Data Cor­rup­tion by 10x Test Escapes Threat­ens Reli­able Com­put­ing. arXiv:2508.01786.
913. Ghe­o­rghe Comanici, Eric Bieber, Mike Schaek­er­mann et al. (2025). Gemini 2.5: Push­ing the Fron­tier with Advanced Rea­son­ing, Mul­ti­modal­ity, Long Con­text, and Next Gen­er­a­tion Agen­tic Capa­bil­i­ties. arXiv:2507.06261.
914. Jef­frey Ma, Hengzhi Pei, Leonard Lausen and George Karypis (2025). Under­stand­ing Silent Data Cor­rup­tion in LLM Train­ing. arXiv:2502.12340.
915. Wei An, Xiao Bi, Guant­ing Chen et al. (2024). Fire-Flyer AI-HPC: A Cost-Effec­tive Soft­ware-Hard­ware Co-Design for Deep Learn­ing. arXiv:2408.14158.
916. Harish Dat­ta­traya Dixit, Laura Boyle, Gau­tham Vunnam et al. (2022). Detect­ing silent data cor­rup­tions in the wild. arXiv:2203.08989.
917. Anton Altenbernd, Philipp Wies­ner and Odej Kao (2026). Explor­ing Silent Data Cor­rup­tion as a Reli­a­bil­ity Chal­lenge in LLM Train­ing. arXiv:2604.00726.
918. Zhipeng Xia, Hao­tian Xu, Siyu Yun et al. (2026). TrainSDC: Char­ac­ter­iz­ing and Mit­i­gat­ing Silent Data Cor­rup­tion in Large Lan­guage Model Train­ing. arXiv:2608.30769.
919. Jinkun Lin, Ziheng Jiang, Zuquan Song et al. (2025). Under­stand­ing Strag­glers in Large Model Train­ing Using What-if Anal­y­sis. arXiv:2505.05713.
920. Tianyuan Wu, Wei Wang, Ying­hao Yu et al. (2024). FALCON: Pin­point­ing and Mit­i­gat­ing Strag­glers for Large-Scale Hybrid-Par­al­lel Train­ing. arXiv:2410.12588.
921. Marco Kurzyn­ski, Shaizeen Aga and Di Wu (2025). Lit Sil­i­con: A Case Where Ther­mal Imbal­ance Cou­ples Con­cur­rent Exe­cu­tion in Mul­ti­ple GPUs. arXiv:2511.09861.
922. Guan­liang Liu, Abhi­nan­dan Patni, Con­gzhu Lin et al. (2026). Guard: Scal­able Strag­gler Detec­tion and Node Health Man­age­ment for Large-Scale Train­ing. arXiv:2605.17879.
923. Yifan Xiong, Yuting Jiang, Ziyue Yang et al. (2024). SuperBench: Improv­ing Cloud AI Infra­struc­ture Reli­a­bil­ity with Proac­tive Val­i­da­tion. arXiv:2402.06194.
924. Borui Wan, Mingji Han, Yiyao Sheng et al. (2024). ByteCheckpoint: A Uni­fied Check­point­ing System for Large Foun­da­tion Model Devel­op­ment. arXiv:2407.20143.
925. PyTorch Con­trib­u­tors (2026). Dis­trib­uted Check­point - torch.dis­trib­uted.check­point. PyTorch 2.15 doc­u­men­ta­tion.
926. Colin Gaffney, Shu­tong Li, Daniel Ng et al. (2026). Orbax: Dis­trib­uted Check­point­ing with JAX. arXiv:2605.23066.
927. Lucas Pasqualin, Less Wright, Iris Zhang et al. (2024). Reduc­ing Model Check­point­ing Times by Over 10x with PyTorch Dis­trib­uted Asyn­chro­nous Check­point­ing. PyTorch blog.
928. Jayashree Mohan, Amar Phan­ishayee and Vijay Chi­dambaram (2021). CheckFreq: Fre­quent, Fine-Grained DNN Check­point­ing. FAST 2021.
929. Zhuang Wang, Zhen Jia, Shuai Zheng et al. (2023). GEMINI: Fast Fail­ure Recov­ery in Dis­trib­uted Train­ing with In-Memory Check­points. Pro­ceed­ings of the 29th Sym­po­sium on Oper­at­ing Sys­tems Prin­ci­ples.
930. Tan­maey Gupta, San­jeev Krish­nan, Rit­u­raj Kumar et al. (2024). Just-In-Time Check­point­ing: Low Cost Error Recov­ery from Deep Learn­ing Train­ing Fail­ures. Pro­ceed­ings of the Nine­teenth Euro­pean Con­fer­ence on Com­puter Sys­tems.
931. John W. Young (1974). A first order approx­i­ma­tion to the opti­mum check­point inter­val. Com­mu­ni­ca­tions of the ACM.
932. J.T. Daly (2006). A higher order esti­mate of the opti­mum check­point inter­val for restart dumps. Future Gen­er­a­tion Com­puter Sys­tems.
933. Vaib­hav Singh and Daniel Her­ring­ton (2024). Intro­duc­ing ML Pro­duc­tiv­ity Good­put: a metric to mea­sure AI system effi­ciency. Google Cloud Blog.
934. Dae­myung Kang, Eunjin Hwang, Han­jeong Lee et al. (2026). From Detec­tion to Recov­ery: Oper­a­tional Anal­y­sis on LLM Pre-train­ing with 504 GPUs. arXiv:2605.09370.
935. Imbue (2024). From bare metal to a 70B model: infra­struc­ture set-up and scripts. Imbue research blog.
936. NVIDIA (2025). DCGM Diag­nos­tics. NVIDIA DCGM doc­u­men­ta­tion.
937. Tao He, Xue Li, Zhibin Wang et al. (2023). Uni­cron: Econ­o­miz­ing Self-Heal­ing LLM Train­ing at Scale. arXiv:2401.00134.
938. PyTorch Con­trib­u­tors (2025). Dis­trib­uted com­mu­ni­ca­tion pack­age - torch.dis­trib­uted. PyTorch 2.9 doc­u­men­ta­tion.
939. Omkar Salpekar, Rohan Varma, Kenny Yu et al. (2026). Train­ing LLMs with Fault Tol­er­ant HSDP on 100,000 GPUs. arXiv:2602.00277.
940. PyTorch Con­trib­u­tors (2026). torchrun (Elas­tic Launch). PyTorch 2.15 doc­u­men­ta­tion.
941. Tris­tan Rice and Howard Huang (2025). Fault Tol­er­ant Llama: train­ing with 2000 syn­thetic fail­ures every ~15 sec­onds and no check­points on Crusoe L40S. PyTorch blog.
942. Zhen­qian Chen, Bao­quan Zhong, Xiang Li et al. (2025). Role-Based Fault Tol­er­ance System for LLM RL Post-Train­ing. arXiv:2512.22492.
943. Andrej Karpa­thy (2019). A Recipe for Train­ing Neural Net­works. Blog post.
944. NVIDIA (2026). Ana­lyz­ing Xid Errors with the Xid Cat­a­log. NVIDIA doc­u­men­ta­tion.
945. Stas Bekman and BigScience Work­shop (2022). The final train­ing: Trials and tribu­la­tions during the 176B 250k multi-lin­gual train­ing. big­science-work­shop/big­science GitHub repos­i­tory (BLOOM tr11⁠–⁠176B-ml chron­i­cles).
946. Yang­tao Deng, Xiang Shi, Zhuo Jiang et al. (2024). Minder: Faulty Machine Detec­tion for Large-scale Dis­trib­uted Model Train­ing. arXiv:2411.01791.
947. PyTorch Con­trib­u­tors (2025). Auto­matic dif­fer­en­ti­a­tion pack­age - torch.auto­grad. PyTorch 2.9 doc­u­men­ta­tion.
948. Yuxuan Jiang, Ziming Zhou, Boyu Xu et al. (2025). Train­ing with Con­fi­dence: Catch­ing Silent Errors in Deep Learn­ing Train­ing with Auto­mated Proac­tive Checks. arXiv:2506.14813.
949. PyTorch Con­trib­u­tors (2024). Repro­ducibil­ity. PyTorch 2.9 doc­u­men­ta­tion.
950. Chirag Pandya and Junjie Wang (2025). Flight Recorder for Debug­ging Stuck Jobs. PyTorch Tuto­ri­als.
951. Ben Fred­er­ick­son (2018). py-spy: Sam­pling pro­filer for Python pro­grams. GitHub repos­i­tory.
952. NVIDIA (2026). Envi­ron­ment Vari­ables — NCCL 2.32.3 doc­u­men­ta­tion. NCCL User Guide.
953. Jiasheng Zhou, Long­bin Zeng, Clavis Chen et al. (2026). ARGUS: Pro­duc­tion-Scale Trac­ing and Per­for­mance Diag­no­sis for over 10,000-GPU Clus­ters. arXiv:2606.20374.
954. Yida Gu, Fakang Wang, Jian­hao Fu et al. (2026). CCL-D: A High-Pre­ci­sion Diag­nos­tic System for Slow and Hang Anom­alies in Large-Scale Model Train­ing. arXiv:2605.04478.
955. Zhihan Jiang, Junjie Huang, Zhuang­bin Chen et al. (2025). L4: Diag­nos­ing Large-scale LLM Train­ing Fail­ures via Auto­mated Log Anal­y­sis. arXiv:2503.20263.
956. Shivam Raikun­dalia (2026). PyTorch Pro­filer. PyTorch Tuto­ri­als.
957. Meta Plat­forms (2024). Holis­tic Trace Anal­y­sis. HTA doc­u­men­ta­tion.
958. NVIDIA (2026). User Guide - Nsight Sys­tems. NVIDIA doc­u­men­ta­tion.
959. The SGLang Team (2025). Towards Deter­min­is­tic Infer­ence in SGLang and Repro­ducible RL Train­ing. LMSYS Org blog.
960. Shiju Zhao, Jiacheng Yang, Qihang Chen et al. (2026). CoRun: Pad­ding is Simple and Effi­cient for Deter­min­is­tic LLM Infer­ence. arXiv:2608.14376.
961. Cecilia Sum­mers and Michael J. Din­neen (2021). Non­de­ter­min­ism and Insta­bil­ity in Neural Net­work Opti­miza­tion. arXiv:2103.04514.
962. Aurick Qiao, Sang Keun Choe, Suhas Jayaram Sub­ra­manya et al. (2020). Pollux: Co-adap­tive Clus­ter Sched­ul­ing for Good­put-Opti­mized Deep Learn­ing. arXiv:2008.12260.
963. Vol­cano project (2026). Intro­duc­tion | Vol­cano. Vol­cano doc­u­men­ta­tion.
964. SchedMD (2025). Slurm Work­load Man­ager - Gang Sched­ul­ing. Slurm doc­u­men­ta­tion.
965. Arnab Choud­hury, Yang Wang, Tuomas Pelko­nen et al. (2024). MAST: Global Sched­ul­ing of ML Train­ing across Geo-Dis­trib­uted Dat­a­cen­ters at Hyper­scale. OSDI 2024.
966. Paul Barham, Aakanksha Chowd­h­ery, Jeff Dean et al. (2022). Path­ways: Asyn­chro­nous Dis­trib­uted Dataflow for ML. arXiv:2203.12533.
967. SchedMD (2025). Slurm Work­load Man­ager - Over­view. Slurm doc­u­men­ta­tion.
968. SchedMD (2025). Slurm Work­load Man­ager - Pre­emp­tion. Slurm doc­u­men­ta­tion.
969. SchedMD (2025). Slurm Work­load Man­ager - slurm.conf. Slurm doc­u­men­ta­tion.
970. Kuber­netes SIGs (2025). JobSet: a k8s native API for dis­trib­uted ML train­ing and HPC work­loads. GitHub repos­i­tory.
971. MLflow (2025). ML Exper­i­ment Track­ing | MLflow. MLflow doc­u­men­ta­tion.
972. Hydra project (2025). Get­ting started | Hydra. Hydra doc­u­men­ta­tion.
973. Weights & Biases (2026). Arti­facts over­view - Weights & Biases Docs. W&B doc­u­men­ta­tion.
974. TensorFlow (2025). TensorBoard | TensorFlow. TensorFlow doc­u­men­ta­tion.
975. WebDataset con­trib­u­tors (2025). web­dataset/web­dataset: A high-per­for­mance Python-based I/O system for large (and small) deep learn­ing prob­lems, with strong sup­port for PyTorch. GitHub repos­i­tory.
976. MosaicML (2025). Stream­ing. MosaicML Stream­ing doc­u­men­ta­tion.
977. PyTorch (2025). State­ful DataLoader — TorchData doc­u­men­ta­tion. TorchData doc­u­men­ta­tion.
978. MLflow (2025). ML Model Reg­istry | MLflow. MLflow doc­u­men­ta­tion.
979. D. Scul­ley, Gary Holt, Daniel Golovin et al. (2015). Hidden Tech­ni­cal Debt in Machine Learn­ing Sys­tems. NeurIPS 2015.
980. Eric Breck, Shan­qing Cai, Eric Nielsen, Michael Salib and D. Scul­ley (2017). The ML Test Score: A Rubric for ML Pro­duc­tion Readi­ness and Tech­ni­cal Debt Reduc­tion. IEEE Inter­na­tional Con­fer­ence on Big Data 2017.
981. NVIDIA (2025). Met­rics | NVIDIA NIM Bench­mark­ing Doc­u­men­ta­tion. NVIDIA NIM doc­u­men­ta­tion.
982. kipply (2022). Trans­former Infer­ence Arith­metic. Blog post.
983. Ted Zadouri, Hubert Strauss and Tri Dao (2025). Hard­ware-Effi­cient Atten­tion for Fast Decod­ing. arXiv:2505.21487.
984. Pol G. Recasens, Ferran Agullo, Yue Zhu et al. (2025). Mind the Memory Gap: Unveil­ing GPU Bot­tle­necks in Large-Batch LLM Infer­ence. arXiv:2503.08311.
985. Megha Agar­wal, Asfand­yar Qureshi, Nikhil Sar­dana et al. (2023). LLM Infer­ence Per­for­mance Engi­neer­ing: Best Prac­tices. Blog post.
986. Pengfei Zuo, Huimin Lin, Junbo Deng et al. (2025). Serv­ing Large Lan­guage Models on Huawei CloudMatrix384. arXiv:2506.12708.
987. Apple (2026). MacBook Air - Tech Specs. Apple prod­uct page (accessed 2026-11-03).
988. Yinmin Zhong, Shengyu Liu, Junda Chen et al. (2024). DistServe: Dis­ag­gre­gat­ing Pre­fill and Decod­ing for Good­put-opti­mized Large Lan­guage Model Serv­ing. arXiv:2401.09670.
989. Amey Agrawal, Anmol Agar­wal, Nitin Kedia et al. (2024). Etalon: Holis­tic Per­for­mance Eval­u­a­tion Frame­work for LLM Infer­ence Sys­tems. arXiv:2407.07000.
990. Jiachen Liu, Jae-Won Chung, Zhiyu Wu et al. (2024). Andes: Defin­ing and Enhanc­ing Qual­ity-of-Expe­ri­ence in LLM-Based Text Stream­ing Ser­vices. arXiv:2404.16283.
991. StepFun, Bin Wang, Bojun Wang et al. (2025). Step-3 is Large yet Afford­able: Model-system Co-design for Cost-effec­tive Decod­ing. arXiv:2507.19427.
992. DeepSeek (2026). Models & Pric­ing | DeepSeek API Docs. DeepSeek API doc­u­men­ta­tion (accessed 2026-11-03).
993. Connor Holmes, Masahiro Tanaka, Michael Wyatt et al. (2024). DeepSpeed-FastGen: High-through­put Text Gen­er­a­tion for LLMs via MII and DeepSpeed-Infer­ence. arXiv:2401.08671.
994. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo­jeong Kim and Byung-Gon Chun (2022). Orca: A Dis­trib­uted Serv­ing System for Trans­former-Based Gen­er­a­tive Models. OSDI 2022.
995. NVIDIA (2026). Over­view - TensorRT LLM. TensorRT-LLM doc­u­men­ta­tion.
996. vLLM Team (2025). Opti­miza­tion and Tuning - vLLM. vLLM doc­u­men­ta­tion.
997. Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachan­dran Ramjee and Ashish Panwar (2024). vAttention: Dynamic Memory Man­age­ment for Serv­ing LLMs with­out PagedAttention. arXiv:2405.04437.
998. Woosuk Kwon, Zhuo­han Li, Siyuan Zhuang et al. (2023). vLLM: Easy, Fast, and Cheap LLM Serv­ing with PagedAttention. vLLM Blog.
999. vLLM Team (2025). Auto­matic Prefix Caching - vLLM. vLLM doc­u­men­ta­tion.
1000. vLLM Team (2025). vLLM V1: A Major Upgrade to vLLM's Core Archi­tec­ture. vLLM Blog.
1001. Lian­min Zheng, Liang­sheng Yin, Zhiqiang Xie et al. (2023). SGLang: Effi­cient Exe­cu­tion of Struc­tured Lan­guage Model Pro­grams. arXiv:2312.07104.
1002. Yuhan Liu, Yihua Cheng, Jiayi Yao et al. (2025). LMCache: An Effi­cient KV Cache Layer for Enter­prise-Scale LLM Infer­ence. arXiv:2510.09665.
1003. Ying Sheng, Shiyi Cao, Dacheng Li et al. (2023). S-LoRA: Serv­ing Thou­sands of Con­cur­rent LoRA Adapters. arXiv:2311.03285.
1004. Amey Agrawal, Nitin Kedia, Ashish Panwar et al. (2024). Taming Through­put-Latency Trade­off in LLM Infer­ence with Sarathi-Serve. arXiv:2403.02310.
1005. Amey Agrawal, Ashish Panwar, Jayashree Mohan et al. (2023). SARATHI: Effi­cient LLM Infer­ence by Pig­gy­back­ing Decodes with Chun­ked Pre­fills. arXiv:2308.16369.
1006. Ruoyu Qin, Zhem­ing Li, Weiran He et al. (2024). Moon­cake: A KVCache-cen­tric Dis­ag­gre­gated Archi­tec­ture for LLM Serv­ing. arXiv:2407.00079.
1007. Junda Chen, Yong­hao Zhuang and Hao Zhang (2025). Dis­ag­gre­gated Infer­ence: 18 Months Later. Hao AI Lab blog.
1008. Chao Wang, Pengfei Zuo, Zhangyu Chen et al. (2025). Pre­fill-Decode Aggre­ga­tion or Dis­ag­gre­ga­tion? Uni­fy­ing Both for Good­put-Opti­mized LLM Serv­ing. arXiv:2508.01989.
1009. SGLang Team (2025). Wel­come to SGLang - SGLang Doc­u­men­ta­tion. SGLang doc­u­men­ta­tion.
1010. Hug­ging Face (2025). Text Gen­er­a­tion Infer­ence. Hug­ging Face doc­u­men­ta­tion.
1011. Georgi Gerganov and ggml-org con­trib­u­tors (2023). ggml-org/llama.cpp: LLM infer­ence in C/C++. GitHub repos­i­tory.
1012. NVIDIA (2025). ai-dynamo/dynamo: A Dat­a­cen­ter Scale Dis­trib­uted Infer­ence Serv­ing Frame­work. GitHub repos­i­tory.
1013. Yao Fu, Leyang Xue, Yeqi Huang et al. (2024). ServerlessLLM: Low-Latency Server­less Infer­ence for Large Lan­guage Models. arXiv:2401.14351.
1014. Ruidong Zhu, Ziheng Jiang, Chao Jin et al. (2025). MegaScale-Infer: Serv­ing Mix­ture-of-Experts at Scale with Dis­ag­gre­gated Expert Par­al­lel­ism. arXiv:2504.02263.
1015. The SGLang Team (2025). Deploy­ing DeepSeek with PD Dis­ag­gre­ga­tion and Large-Scale Expert Par­al­lel­ism on 96 H100 GPUs. LMSYS Org blog.
1016. Mitchell Stern, Noam Shazeer and Jakob Uszko­r­eit (2018). Block­wise Par­al­lel Decod­ing for Deep Autore­gres­sive Models. arXiv:1811.03115.
1017. Yaniv Leviathan, Matan Kalman and Yossi Matias (2022). Fast Infer­ence from Trans­form­ers via Spec­u­la­tive Decod­ing. arXiv:2211.17192.
1018. Char­lie Chen, Sebas­tian Borgeaud, Geof­frey Irving et al. (2023). Accel­er­at­ing Large Lan­guage Model Decod­ing with Spec­u­la­tive Sam­pling. arXiv:2302.01318.
1019. Amirhos­sein Bozorgkhoo and Igor Moly­bog (2026). Spec­u­la­tive Decod­ing Scal­ing Laws (SDSL): Through­put Opti­miza­tion Made Simple. arXiv:2603.11053.
1020. Yuhui Li, Fangyun Wei, Chao Zhang and Hongyang Zhang (2024). EAGLE: Spec­u­la­tive Sam­pling Requires Rethink­ing Fea­ture Uncer­tainty. arXiv:2401.15077.
1021. Yuhui Li, Fangyun Wei, Chao Zhang and Hongyang Zhang (2025). EAGLE-3: Scal­ing up Infer­ence Accel­er­a­tion of Large Lan­guage Models via Train­ing-Time Test. arXiv:2503.01840.
1022. Tianle Cai, Yuhong Li, Zhengyang Geng et al. (2024). Medusa: Simple LLM Infer­ence Accel­er­a­tion Frame­work with Mul­ti­ple Decod­ing Heads. arXiv:2401.10774.
1023. Yuhui Li, Fangyun Wei, Chao Zhang and Hongyang Zhang (2024). EAGLE-2: Faster Infer­ence of Lan­guage Models with Dynamic Draft Trees. arXiv:2406.16858.
1024. Yichao Fu, Peter Bailis, Ion Stoica and Hao Zhang (2024). Break the Sequen­tial Depen­dency of LLM Infer­ence Using Looka­head Decod­ing. arXiv:2402.02057.
1025. Rana­joy Sad­hukhan, Jian Chen, Zhuom­ing Chen et al. (2024). MagicDec: Break­ing the Latency-Through­put Trade­off for Long Con­text Gen­er­a­tion with Spec­u­la­tive Decod­ing. arXiv:2408.11049.
1026. Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh and Acyr Locatelli (2026). Why Spec­u­la­tive Decod­ing Works Better Than Pre­dicted on Sparse MoE Models. arXiv:2610.38981.
1027. Yongchao Long, Chun­rui Zheng, Yuxuan Zhou et al. (2026). When Does Better Accep­tance Pay? A Con­trolled Study of Spec­u­la­tive Decod­ing. arXiv:2610.24993.
1028. Vage Egiazar­ian, Roberto L. Castro, Denis Kuznedelev et al. (2025). Bridg­ing the Gap Between Prom­ise and Per­for­mance for Microscal­ing FP4 Quan­ti­za­tion. arXiv:2509.23202.
1029. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettle­moyer (2022). LLM.int8(): 8-bit Matrix Mul­ti­pli­ca­tion for Trans­form­ers at Scale. arXiv:2208.07339.
1030. Guangx­uan Xiao, Ji Lin, Mick­ael Seznec et al. (2022). SmoothQuant: Accu­rate and Effi­cient Post-Train­ing Quan­ti­za­tion for Large Lan­guage Models. arXiv:2211.10438.
1031. Saleh Ashk­boos, Amirkeivan Mohtashami, Max­i­m­il­ian L. Croci et al. (2024). QuaRot: Out­lier-Free 4-Bit Infer­ence in Rotated LLMs. arXiv:2404.00456.
1032. Elias Fran­tar, Saleh Ashk­boos, Torsten Hoe­fler and Dan Alis­tarh (2022). GPTQ: Accu­rate Post-Train­ing Quan­ti­za­tion for Gen­er­a­tive Pre-trained Trans­form­ers. arXiv:2210.17323.
1033. Ji Lin, Jiaming Tang, Hao­tian Tang et al. (2023). AWQ: Acti­va­tion-aware Weight Quan­ti­za­tion for LLM Com­pres­sion and Accel­er­a­tion. arXiv:2306.00978.
1034. Eldar Kurtic, Alexan­dre Mar­ques, Shubhra Pandit, Mark Kurtz and Dan Alis­tarh (2024). "Give Me BF16 or Give Me Death"? Accu­racy-Per­for­mance Trade-Offs in LLM Quan­ti­za­tion. arXiv:2411.02355.
1035. Yujun Lin, Hao­tian Tang, Shang Yang et al. (2024). QServe: W4A8KV4 Quan­ti­za­tion and System Co-design for Effi­cient LLM Serv­ing. arXiv:2405.04532.
1036. Mart van Baalen, Andrey Kuzmin, Suparna S Nair et al. (2023). FP8 versus INT8 for effi­cient deep learn­ing infer­ence. arXiv:2303.17951.
1037. Abhi­nav Dutta, San­jeev Krish­nan, Nipun Kwatra and Ramachan­dran Ramjee (2024). Accu­racy is Not All You Need. arXiv:2407.09141.
1038. Jiayi Yuan, Hongyi Liu, Shaochen Zhong et al. (2024). KV Cache Com­pres­sion, But What Must We Give in Return? A Com­pre­hen­sive Bench­mark of Long Con­text Capa­ble Approaches. arXiv:2407.01527.
1039. Jiashi Li, Shengyu Liu and Yuan­hang Sun (2025). FlashMLA: Effi­cient Multi-head Latent Atten­tion Ker­nels. GitHub repos­i­tory.
1040. Zirui Liu, Jiayi Yuan, Hongye Jin et al. (2024). KIVI: A Tuning-Free Asym­met­ric 2bit Quan­ti­za­tion for KV Cache. arXiv:2402.02750.
1041. Cole­man Hooper, Sehoon Kim, Hiva Moham­madzadeh et al. (2024). KVQuant: Towards 10 Mil­lion Con­text Length LLM Infer­ence with KV Cache Quan­ti­za­tion. arXiv:2401.18079.
1042. Zhenyu Zhang, Ying Sheng, Tianyi Zhou et al. (2023). H₂O: Heavy-Hitter Oracle for Effi­cient Gen­er­a­tive Infer­ence of Large Lan­guage Models. arXiv:2306.14048.
1043. Yuhong Li, Ying­bing Huang, Bowen Yang et al. (2024). SnapKV: LLM Knows What You are Look­ing for Before Gen­er­a­tion. arXiv:2404.14469.
1044. Jiaming Tang, Yilong Zhao, Kan Zhu et al. (2024). Quest: Query-Aware Spar­sity for Effi­cient Long-Con­text LLM Infer­ence. arXiv:2406.10774.
1045. Guangx­uan Xiao, Jiaming Tang, Jing­wei Zuo et al. (2024). DuoAttention: Effi­cient Long-Con­text LLM Infer­ence with Retrieval and Stream­ing Heads. arXiv:2410.10819.
1046. Tri Dao, Daniel Haziza, Fran­cisco Massa and Grig­ory Sizov (2023). Flash-Decod­ing for long-con­text infer­ence. Stan­ford CRFM blog.
1047. Zihao Ye, Lequn Chen, Rui­hang Lai et al. (2025). FlashInfer: Effi­cient and Cus­tom­iz­a­ble Atten­tion Engine for LLM Infer­ence Serv­ing. arXiv:2501.01005.
1048. Ari Holtz­man, Jan Buys, Li Du, Max­well Forbes and Yejin Choi (2019). The Curi­ous Case of Neural Text Degen­er­a­tion. arXiv:1904.09751.
1049. Angela Fan, Mike Lewis and Yann Dauphin (2018). Hier­ar­chi­cal Neural Story Gen­er­a­tion. arXiv:1805.04833.
1050. Minh Nhat Nguyen, Andrew Baker, Clement Neo et al. (2024). Turn­ing Up the Heat: Min-p Sam­pling for Cre­ative and Coher­ent LLM Out­puts. arXiv:2407.01082.
1051. Chenxia Tang, Jianchun Liu, Hongli Xu and Liusheng Huang (2024). Top-nσ: Not All Logits Are You Need. arXiv:2411.07641.
1052. John Hewitt, Christo­pher D. Man­ning and Percy Liang (2022). Trun­ca­tion Sam­pling as Lan­guage Model Desmooth­ing. arXiv:2210.15191.
1053. Yifan Song, Guoyin Wang, Sujian Li and Bill Yuchen Lin (2024). The Good, The Bad, and The Greedy: Eval­u­a­tion of LLMs Should Not Ignore Non-Deter­min­ism. arXiv:2407.10457.
1054. Matthew Renze and Erhan Guven (2024). The Effect of Sam­pling Tem­per­a­ture on Prob­lem Solv­ing in Large Lan­guage Models. arXiv:2402.05201.
1055. Rylan Scha­ef­fer, Joshua Kazdan and Yegor Denisov-Blanch (2025). Min-p, Max Exag­ger­a­tion: A Crit­i­cal Anal­y­sis of Min-p Sam­pling in Lan­guage Models. arXiv:2506.13681.
1056. Bran­don T. Willard and Rémi Louf (2023). Effi­cient Guided Gen­er­a­tion for Large Lan­guage Models. arXiv:2307.09702.
1057. Kanghee Park, Jiayu Wang, Taylor Berg-Kirk­patrick, Nadia Polikar­pova and Loris D'Antoni (2024). Gram­mar-Aligned Decod­ing. arXiv:2405.21047.
1058. Luca Beurer-Kell­ner, Marc Fis­cher and Martin Vechev (2024). Guid­ing LLMs The Right Way: Fast, Non-Inva­sive Con­strained Gen­er­a­tion. arXiv:2403.06988.
1059. Saibo Geng, Hudson Cooper, Michał Moskal et al. (2025). JSONSchemaBench: A Rig­or­ous Bench­mark of Struc­tured Out­puts for Lan­guage Models. arXiv:2501.10868.
1060. Yixin Dong, Char­lie F. Ruan, Yaxing Cai et al. (2024). XGrammar: Flex­i­ble and Effi­cient Struc­tured Gen­er­a­tion Engine for Large Lan­guage Models. arXiv:2411.15100.
1061. OpenAI (2021). human-eval/human_eval/exe­cu­tion.py at master · openai/human-eval. GitHub repos­i­tory.
1062. Chris Lu, Cong Lu, Robert Tjarko Lange et al. (2024). The AI Sci­en­tist: Towards Fully Auto­mated Open-Ended Sci­en­tific Dis­cov­ery. arXiv:2408.06292.
1063. OpenAI (2024). OpenAI o1 System Card (Sep­tem­ber 12, 2024). System card.
1064. Max McGuinness, Mikaela Grace, Jiri De Jonghe, Jake Eaton and Abel Rib­bink (2026). How we con­tain Claude across prod­ucts. Anthropic Engi­neer­ing blog.
1065. Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir and Nick Judd (2025). SandboxEval: Towards Secur­ing Test Envi­ron­ment for Untrusted Code. arXiv:2504.00018.
1066. Kimi Team, Tong­tong Bai, Yifan Bai et al. (2026). Kimi K3: Open Fron­tier Intel­li­gence. arXiv:2607.24653.
1067. Simhadri Podala Narasimha (2026). Con­tain­ing the Auton­o­mous Oper­a­tor: A Defense-in-Depth Frame­work and Ref­er­ence Archi­tec­ture for Secur­ing AI Agents on Kuber­netes. arXiv:2610.02861.
1068. Simon Willi­son (2025). The lethal tri­fecta for AI agents: pri­vate data, untrusted con­tent, and exter­nal com­mu­ni­ca­tion. Blog post.
1069. gVisor Authors (2024). Secu­rity Model. gVisor doc­u­men­ta­tion.
1070. Linux man-pages project (2026). names­paces(7) - Linux manual page. Linux man-pages.
1071. Tejun Heo (2015). Con­trol Group v2. The Linux Kernel doc­u­men­ta­tion.
1072. Docker Inc. (2025). Docker Engine secu­rity. Docker doc­u­men­ta­tion.
1073. Linux man-pages project (2026). capa­bil­i­ties(7) - Linux manual page. Linux man-pages.
1074. The Kuber­netes Authors (2026). Pod Secu­rity Stan­dards. Kuber­netes doc­u­men­ta­tion.
1075. Linux man-pages project (2026). sec­comp(2) - Linux manual page. Linux man-pages.
1076. Docker Inc. (2026). Sec­comp secu­rity pro­files for Docker. Docker doc­u­men­ta­tion.
1077. Linux man-pages project (2026). user_names­paces(7) - Linux manual page. Linux man-pages.
1078. The Kuber­netes Authors (2026). User Names­paces. Kuber­netes doc­u­men­ta­tion.
1079. Linux man-pages project (2026). land­lock(7) - Linux manual page. Linux man-pages.
1080. Steven R Brandt, Max Morris, Patrick Diehl et al. (2025). Lock­ing Down Sci­ence Gate­ways with Land­lock and Sec­comp. arXiv:2509.18548.
1081. Open Con­tainer Ini­tia­tive (2025). Linux Con­tainer Con­fig­u­ra­tion. GitHub (open­con­tain­ers/run­time-spec/blob/main/config-linux.md).
1082. Open Con­tainer Ini­tia­tive (2024). Open Con­tainer Ini­tia­tive Run­time Spec­i­fi­ca­tion. GitHub repos­i­tory.
1083. gVisor Authors (2024). What is gVisor?. gVisor doc­u­men­ta­tion.
1084. Kata Con­tain­ers (2026). Kata Con­tain­ers Archi­tec­ture. GitHub (kata-con­tain­ers/kata-con­tain­ers/blob/main/docs/design/archi­tec­ture/README.md).
1085. Maysara Alhindi and Joseph Hal­lett (2025). Play­ing in the Sand­box: A Study on the Usabil­ity of Sec­comp. arXiv:2506.10234.
1086. Muru­giah Soup­paya, John Morello and Karen Scar­fone (2017). Appli­ca­tion con­tainer secu­rity guide. National Insti­tute of Stan­dards and Tech­nol­ogy.
1087. Alexan­dru Agache, Marc Brooker, Andreea Flo­rescu et al. (2020). Fire­cracker: Light­weight Vir­tu­al­iza­tion for Server­less Appli­ca­tions. 17th USENIX Sym­po­sium on Net­worked Sys­tems Design and Imple­men­ta­tion (NSDI 20).
1088. Ethan G. Young, Pengfei Zhu, Tyler Caraza-Harter, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau (2019). The True Cost of Con­tain­ing: A gVisor Case Study. HotCloud 2019.
1089. gVisor Authors (2024). Per­for­mance Guide. gVisor doc­u­men­ta­tion.
1090. gVisor Authors (2024). Plat­form Guide. gVisor doc­u­men­ta­tion.
1091. Ayush Ranjan (2023). Faster filesys­tem access with Directfs. gVisor blog.
1092. Ayush Ranjan (2023). Rootfs Over­lay. gVisor blog.
1093. Fire­cracker Project (2026). Spec­i­fi­ca­tion. GitHub (fire­cracker-microvm/fire­cracker/blob/main/SPECIFICATION.md).
1094. Marc Brooker, Adrian Costin Catangiu, Mike Danilov et al. (2021). Restor­ing Unique­ness in MicroVM Snap­shots. arXiv:2102.12892.
1095. Dmitrii Ustiu­gov, Plamen Petrov, Marios Kogias, Edouard Bugnion and Boris Grot (2021). Bench­mark­ing, Anal­y­sis, and Opti­miza­tion of Server­less Func­tion Snap­shots. arXiv:2101.09355.
1096. Kuber­netes SIG Apps (2026). agent-sand­box. GitHub repos­i­tory.
1097. WebAssembly Com­mu­nity Group (2024). WASI.dev. WASI project site.
1098. Simon Shillaker and Peter Piet­zuch (2020). Faasm: Light­weight Iso­la­tion for Effi­cient State­ful Server­less Com­put­ing. arXiv:2002.09344.
1099. Abhi­nav Jangda, Bobby Powers, Emery Berger and Arjun Guha (2019). Not So Fast: Ana­lyz­ing the Per­for­mance of WebAssembly vs. Native Code. arXiv:1901.09056.
1100. Con­tain­ers project (2024). con­tain­ers/bub­blewrap: Low-level unpriv­i­leged sand­box­ing tool used by Flat­pak and sim­i­lar projects. GitHub repos­i­tory.
1101. Google (2024). google/nsjail: A light­weight process iso­la­tion tool that uti­lizes Linux names­paces, cgroups, rlim­its and sec­comp-bpf syscall fil­ters. GitHub repos­i­tory.
1102. SWE-bench team (2024). Docker Setup. SWE-bench doc­u­men­ta­tion.
1103. UK AI Secu­rity Insti­tute (2025). Sand­box­ing – Inspect. Inspect doc­u­men­ta­tion.
1104. Yiming Yan, Hongyang Wang, Wenbo Chen et al. (2026). Sand­man: Build­ing and Oper­at­ing Sand­boxes for Agen­tic Rein­force­ment Learn­ing on Shared Kuber­netes. arXiv:2610.39711.
1105. The Kuber­netes Authors (2026). Resource Man­age­ment for Pods and Con­tain­ers. Kuber­netes doc­u­men­ta­tion.
1106. The Kuber­netes Authors (2026). Jobs. Kuber­netes doc­u­men­ta­tion.
1107. Ziqian Zhong, Aditi Raghu­nathan and Nicholas Car­lini (2025). ImpossibleBench: Mea­sur­ing LLMs' Propen­sity of Exploit­ing Test Cases. arXiv:2510.20270.
1108. Fire­cracker Project (2026). Fire­cracker Snap­shot­ting. GitHub (fire­cracker-microvm/fire­cracker/blob/main/docs/snap­shot­ting/snap­shot-sup­port.md).
1109. John Yang, Kilian Lieret, Carlos E. Jimenez et al. (2025). SWE-smith: Scal­ing Data for Soft­ware Engi­neer­ing Agents. arXiv:2504.21798.
1110. Ibragim Badert­di­nov, Alexan­der Gol­ubev, Maksim Nekra­she­vich et al. (2025). SWE-rebench: An Auto­mated Pipe­line for Task Col­lec­tion and Decon­tam­i­nated Eval­u­a­tion of Soft­ware Engi­neer­ing Agents. arXiv:2505.20411.
1111. Greg Brock­man, Vicki Cheung, Ludwig Pet­ters­son et al. (2016). OpenAI Gym. arXiv:1606.01540.
1112. Mark Towers, Ariel Kwiatkowski, Jordan Terry et al. (2024). Gym­na­sium: A Stan­dard Inter­face for Rein­force­ment Learn­ing Envi­ron­ments. arXiv:2407.17032.
1113. Xingyao Wang, Boxuan Li, Yufan Song et al. (2024). OpenHands: An Open Plat­form for AI Soft­ware Devel­op­ers as Gen­er­al­ist Agents. arXiv:2407.16741.
1114. Anthropic (2024). Intro­duc­ing the Model Con­text Pro­to­col. Anthropic news.
1115. Model Con­text Pro­to­col (2025). Spec­i­fi­ca­tion. mod­el­con­textpro­to­col.io.
1116. SWE-bench team (2024). Eval­u­a­tion - SWE-bench. SWE-bench doc­u­men­ta­tion.
1117. John Yang, Carlos E. Jimenez, Alexan­der Wettig et al. (2024). SWE-agent: Agent-Com­puter Inter­faces Enable Auto­mated Soft­ware Engi­neer­ing. arXiv:2405.15793.
1118. Flo­rian Brand and Jean-Stanis­las Denain (2025). What skills does SWE-bench Ver­i­fied eval­u­ate?. Epoch AI blog.
1119. Tom Adam­czewski (2025). How to run SWE-bench Ver­i­fied in one hour on one machine. Epoch AI blog.
1120. Thibault Le Sel­l­ier De Chezelles, Maxime Gasse, Alexan­dre Drouin et al. (2024). The BrowserGym Ecosys­tem for Web Agent Research. arXiv:2412.05467.
1121. Harsh Trivedi, Tushar Khot, Mareike Hart­mann et al. (2024). AppWorld: A Con­trol­lable World of Apps and People for Bench­mark­ing Inter­ac­tive Coding Agents. arXiv:2407.18901.
1122. Tian­bao Xie, Mengqi Yuan, Danyang Zhang et al. (2025). Intro­duc­ing OSWorld-Ver­i­fied. XLANG blog.
1123. Edward Lue Chee Lip, Boden Moraski, Tim Knappe et al. (2026). What Makes a Ter­mi­nal-Bench Task Hard? Sep­a­rat­ing Gen­uine Hard­ness from Fake-Hard­ness on an Adju­di­cated Agen­tic Corpus. arXiv:2609.26826.
1124. You Wang, Michael Pradel and Zhongxin Liu (2025). Are "Solved Issues" in SWE-bench Really Solved Cor­rectly? An Empir­i­cal Study. arXiv:2503.15223.
1125. Reem Alei­than, Haoran Xue, Moham­mad Mahdi Moha­jer et al. (2024). SWE-Bench+: Enhanced Coding Bench­mark for LLMs. arXiv:2410.06992.
1126. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Mari­nov (2014). An empir­i­cal anal­y­sis of flaky tests. Pro­ceed­ings of the 22nd ACM SIGSOFT Inter­na­tional Sym­po­sium on Foun­da­tions of Soft­ware Engi­neer­ing.
1127. Dario Amodei, Chris Olah, Jacob Stein­hardt et al. (2016). Con­crete Prob­lems in AI Safety. arXiv:1606.06565.
1128. Joar Skalse, Niko­laus H. R. Howe, Dmitrii Krashenin­nikov and David Krueger (2022). Defin­ing and Char­ac­ter­iz­ing Reward Hack­ing. arXiv:2209.13085.
1129. Anthropic (2025). Claude 3.7 Sonnet System Card. Anthropic system card.
1130. Amit Roth, Ivan Bercovich and Yonathan Efroni (2026). Hack-Ver­i­fi­able Ter­mi­nal Bench: Eval­u­at­ing Reward Hack­ing in Ter­mi­nal Tasks. arXiv:2608.22103.
1131. Mia Taylor, James Chua, Jan Betley, Johannes Treut­lein and Owain Evans (2025). School of Reward Hacks: Hack­ing harm­less tasks gen­er­al­izes to mis­aligned behav­ior in LLMs. arXiv:2508.17511.
1132. Anthropic (2025). Nat­u­ral emer­gent mis­align­ment from reward hack­ing. Anthropic research.
1133. Bowen Baker, Joost Huizinga, Alek­sander Madry et al. (2025). Detect­ing mis­be­hav­ior in fron­tier rea­son­ing models. OpenAI blog (archived).
1134. Marc Brooker, Mike Danilov, Chris Green­wood and Phil Piwonka (2023). On-demand Con­tainer Load­ing in AWS Lambda. arXiv:2305.13162.
1135. Tyler Harter, Bran­don Salmon, Rose Liu, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau (2016). Slacker: Fast Dis­tri­bu­tion with Lazy Docker Con­tain­ers. 14th USENIX Con­fer­ence on File and Stor­age Tech­nolo­gies (FAST 16).
1136. Yun­peng Dong, Jingkai He, Shiqi Liu et al. (2026). DeltaBox: Scal­ing State­ful AI Agents with Mil­lisec­ond-Level Sand­box Check­point/Roll­back. arXiv:2605.22781.
1137. The Kuber­netes Authors (2026). Con­sid­er­a­tions for large clus­ters. Kuber­netes doc­u­men­ta­tion.
1138. Ning Shang, Yifei Liu, Yi Zhu et al. (2025). rStar2-Agent: Agen­tic Rea­son­ing Tech­ni­cal Report. arXiv:2508.20722.
1139. GLM-5-Team, Aohan Zeng, Xin Lv et al. (2026). GLM-5: from Vibe Coding to Agen­tic Engi­neer­ing. arXiv:2602.15763.
1140. Gian Segato (2026). Quan­ti­fy­ing infra­struc­ture noise in agen­tic coding evals. Anthropic Engi­neer­ing blog.
1141. Qwen Team (2025). Qwen3-Coder: Agen­tic Coding in the World. Blog post.
1142. Kairong Luo, Haodong Wen, Shengding Hu et al. (2025). A Multi-Power Law for Loss Curve Pre­dic­tion Across Learn­ing Rate Sched­ules. arXiv:2503.12811.
1143. Jonathan Coles, Ste­fano Schup­pli, Lukas Drescher et al. (2026). An Engi­neer­ing Jour­ney Train­ing Large Lan­guage Models at Scale on Alps: The Aper­tus Expe­ri­ence. arXiv:2604.12973.
1144. Tat­sunori Hashimoto and Percy Liang (2026). Stan­ford CS336 | Lan­guage Mod­el­ing from Scratch. Stan­ford Uni­ver­sity course.
1145. Jacob Austin, Sholto Dou­glas, Roy Frostig et al. (2025). How To Scale Your Model. Google DeepMind (online book).
1146. Andrej Karpa­thy (2022). nanoGPT. GitHub repos­i­tory.
1147. Andrej Karpa­thy (2024). llm.c. GitHub repos­i­tory.
1148. Andrej Karpa­thy (2025). nanochat: The best ChatGPT that $100 can buy. GitHub repos­i­tory.
1149. Stas Bekman (2026). Machine Learn­ing Engi­neer­ing Open Book. GitHub repos­i­tory (open book).
1150. Bingchen Zhao, Despoina Magka, Minqi Jiang et al. (2025). The Auto­mated LLM Speedrun­ning Bench­mark: Repro­duc­ing NanoGPT Improve­ments. arXiv:2506.22419.
1151. Ege Erdil and David Schnei­der-Joseph (2024). Data move­ment limits to fron­tier model train­ing. arXiv:2411.01137.
1152. Mitchell Worts­man, Gabriel Ilharco, Samir Yitzhak Gadre et al. (2022). Model soups: aver­ag­ing weights of mul­ti­ple fine-tuned models improves accu­racy with­out increas­ing infer­ence time. arXiv:2203.05482.