RL environments: a field guide

Rein­force­ment learn­ing (RL) trains a model by let­ting it act, scor­ing what it did and making high-scor­ing behav­ior more likely. For a lan­guage-model agent, one attempt, called a roll­out or episode, is a chain of model calls and tool calls that ends in an answer or a changed system. A train­ing run is mil­lions of them, and every­thing the model learns arrives through one narrow chan­nel: the number the envi­ron­ment returns at the end.

That makes an RL envi­ron­ment a mea­sur­ing instru­ment with an opti­mizer pointed at it. Every roll­out buys infor­ma­tion, and the envi­ron­ment decides whether the pur­chase returns signal, noise or a lie. Most envi­ron­ment-side fail­ures, from reward hack­ing to gains that vanish on re-eval­u­a­tion, are the instru­ment fail­ing in one of six ways. Its tasks can be empty: the policy always or never suc­ceeds, so noth­ing is learned. Its reward can be wrong, paying for some­thing other than the task. The train­ing record can be cor­rupted, no longer match­ing what the policy did. The signal can be coarse, one bit per episode when a full rank­ing was avail­able. The effect can be unread­able, smaller than the eval­u­a­tion's noise floor. And the infor­ma­tion needed to win can be absent, unavail­able to any legal player. This guide calls that list the signal ledger (six losses, plus line 0 for the instru­ment itself) and works through it in order.

The guide is for a tech­ni­cally lit­er­ate reader who wants the mechan­ics: how to spec­ify an envi­ron­ment a lab would trust, where a run will waste com­pute or mis­lead you, and how to judge some­one else's claimed gain. Each sec­tion opens with its ques­tion and closes with a deci­sion rule, and the rules are col­lected in a check­list at the end.

How to read this guide

Four kinds of number. Most num­bers cite a public paper, report, doc­u­men­ta­tion page or leader­board; papers and system cards are cited as they stand, "self-reported" marks a com­pany's claim about its own system in a blog, its doc­u­men­ta­tion or a leader­board it runs, and "reported" a press account. Com­puted num­bers are arith­metic you can redo from a stated for­mula. Toy sim­u­la­tions are small seeded pro­grams that iso­late one mech­a­nism and show the shape of an effect, never its size: a route-plan­ning task pool (Part II), a 0.1M-param­e­ter trans­former learn­ing addi­tion (Parts III to V), a roll­out system with heavy-tailed latency (Part IV) and an order-exe­cu­tion sim­u­la­tor (Part VI). Test­bed mea­sure­ments come from two larger testbeds the author built. Toy and test­bed num­bers are unpub­lished, so no rule rests on them alone; the meth­ods table at the end gives each one's setup, seeds and inter­val method.

Two testbeds. The rec­on­cil­i­a­tion envi­ron­ment (Parts I to III) asks an agent to rec­on­cile an invest­ment fund's book with its cus­to­dian's state­ments and its admin­is­tra­tor's draft val­u­a­tion, find the "breaks" where they dis­agree, post cor­rec­tions and sign off a net asset value within 1 basis point (0.01%). The cross­word-tile-game test­bed¹ (Parts V and VI) is a two-player word game with a shuf­fled bag and racks the oppo­nent cannot see, in which a search player can score every legal move. That makes it a clean instru­ment for luck ceil­ings, dis­til­la­tion from a search oracle, paired eval­u­a­tion, search versus learn­ing, and the limits of self-play.

The terms below are used in exactly these senses through­out, and each is also defined where it first appears.

Part IThe environment is the product

What is an RL envi­ron­ment, why did it become the scarce input to fron­tier train­ing, and what must it prove before anyone trains on it?

Line 0 of the ledger is the instru­ment itself: a bound­ary between what the agent can touch and what grades it, whose bind­ing input is sound tasks, not sand­boxes.

Three sides of one boundary

What exactly does a lab train on? In this guide an envi­ron­ment is a task (or a world that hosts many tasks), the tools and state an agent acts on, and a ver­i­fier that runs out of the agent's reach and turns the fin­ished episode into a reward; when a model does the scor­ing, it is a judge. Think of an exam hall: the can­di­date holds the pen, never the answer key, and the paper is marked after the can­di­date leaves. The pic­ture has three sides (Figure 1).

• Agent side. Typed tools and noth­ing else. Obser­va­tions never carry the reward or any ground truth, because what­ever the agent can read, the opti­mizer will learn to use. A mal­formed call returns an error the model can recover from, as the Model Con­text Pro­to­col (MCP), the common stan­dard for tool calls, asks: "action­able feed­back that lan­guage models can use to self-cor­rect" [17].
• Infra­struc­ture side. Episode con­trol: reset, step, the clock, the call that starts ver­i­fi­ca­tion, and the bank that holds rewards that resolve later. An agent that calls any of these has crossed the bound­ary, and the har­ness logs an integrity event, not an error to retry.
• Ver­i­fier side. The ver­i­fier runs once, after the episode, in its own sand­box with no net­work. It reads the exported final state, both what changed and what should not have, and returns a scalar plus named com­po­nents. Tests, answer key, ref­er­ence solu­tion and scripted hacks live here, never where the agent runs.

OpenEnv, an open envi­ron­ment spec­i­fi­ca­tion hosted by Hug­ging Face, draws the same line: sim­u­la­tion con­trols such as reset and step "are never exposed as MCP tools", because an agent that could reset "would learn that every error is recov­er­able" [18]. One trap: OpenEnv's obser­va­tion type car­ries a reward field for the train­ing loop, so a har­ness that for­wards obser­va­tions ver­ba­tim shows the agent its own score. Strip it at the bound­ary.

When public audits go look­ing, the holes sit in this bound­ary, not in the model or the com­pute. A UC Berke­ley team's exploit agent, BenchJack, scored 73% to 100% on eight widely used agent bench­marks (4,458 tasks) with­out doing the work: agent-writ­ten test hooks made every SWE-bench Ver­i­fied test pass, con­fig­u­ra­tion files exposed WebArena's ref­er­ence answers, and one val­ida­tor gave full marks to an empty object [19]. Their rule: "Iso­late every­thing that scores from every­thing being scored" [20]. The bound­ary moves as models improve. GSO, a code-opti­miza­tion bench­mark, cut its con­tain­ers off from code hosts in July 2026 "after we observed newer models were some­times using arbi­trary tech­niques to fetch upstream com­mits" [21]. And infra­struc­ture must be unreach­able, not merely unlisted: in OpenAI's o1 eval­u­a­tions a cap­ture-the-flag con­tainer failed to start, so o1-pre­view scanned the net­work, found the eval­u­a­tion host's Docker API exposed by a mis­con­fig­u­ra­tion, and started a fresh copy of the con­tainer with the com­mand cat flag.txt to read the flag [22] (cross­ing 6 in Figure 1).

Why now: checkable tasks were never stockpiled

Why did envi­ron­ments, rather than data or com­pute, become the bot­tle­neck? Pre­train­ing scaled on text the web had already writ­ten. After OpenAI's o1 (Sep­tem­ber 2024) and DeepSeek-R1 (Jan­u­ary 2025) showed that reward­ing checked answers in math and code drives rea­son­ing, RL on ver­i­fi­able tasks became the way labs add capa­bil­ity [23] [24]. That con­sumes some­thing nobody had stock­piled: tasks with a reli­able check, set in places that look like real work.

Reports now count envi­ron­ments in the thou­sands and claim hun­dreds of thou­sands, though each counts a dif­fer­ent unit. DeepSeek-V3.2 had an agent syn­the­size 1,827 envi­ron­ments hold­ing 4,417 tasks, inside a post-train­ing budget "exceed­ing 10% of the pre-train­ing cost" [25]; Prime Intel­lect pack­ages about 365,000 tasks from 23 task sets for soft­ware, ter­mi­nal and search work (self-reported) [26]. One train­ing step explains the con­cur­rency. GRPO (group rel­a­tive policy opti­miza­tion), the method DeepSeek-R1 trained with, sam­ples a group of roll­outs for each prompt and scores each against the group's mean. A step of 64 prompts with 8 roll­outs each needs 512 envi­ron­ment instances live at once (com­puted).

None of the plumb­ing is scarce any more: a man­aged sand­box lists at $0.000014 per vCPU-second, about five cents an hour [27], and DeepSeek reports more than 380,000 con­cur­rent sand­boxes [28]. What each of those 512 slots needs is a task whose reward is sound and whose pass rate sits where the cur­rent policy learns; in the words of one of Epoch AI's inter­vie­wees, "Main­tain­ing qual­ity while scal­ing is the number one bot­tle­neck that people see" [29].

What a task is worth

Who pays for envi­ron­ments, and what does one task cost against what it con­sumes? In Sep­tem­ber 2025 The Infor­ma­tion reported that Anthropic had dis­cussed spend­ing more than $1 bil­lion on RL envi­ron­ments over the fol­low­ing year, as Epoch AI's FAQ on the market recounts [29]. Epoch's inter­views put con­tracts at "six to seven fig­ures per quar­ter", tasks at "$200⁠–⁠$2,000 per task", web­site repli­cas ("UI gyms") at around $20k each and a high-fidelity replica of a prod­uct such as Slack at about $300k, with exclu­sive deals "roughly 4⁠–⁠5x" the price [29]. Com­pute will "still dwarf RL envi­ron­ment spend­ing" [29], so price a task by its draw on com­pute. Mech­a­nize puts that at about $480 per train­ing run, assum­ing 500,000 tokens per attempt (its one-year extrap­o­la­tion from about 100,000 when it wrote), 64 attempts and an API price of $15 per mil­lion tokens [30]; at 100,000 tokens the same arith­metic gives about $96, and at a lab's inter­nal cost less (com­puted). Either way, a $50 task whose roll­outs all fail, or all pass, burns two to ten times its price on every run and teaches noth­ing (com­puted). Ven­dors, pro­cure­ment and pric­ing are the sub­ject of the com­pan­ion arti­cle, RL envi­ron­ments as a busi­ness.

Environments are consumables

How long does an envi­ron­ment stay useful? Not long, once some­one trains against it: bench­marks built from the same kind of task fill up on a sched­ule (Figure 2). CVDP, a chip-design bench­mark, launched in June 2025 with state-of-the-art models at "no more than 34% pass@1" (single-attempt suc­cess) on code gen­er­a­tion [31]; thir­teen months later NVIDIA reported a 97.1% aver­age pass rate across nine cat­e­gories for Nemotron 3 Ultra inside an agent that gen­er­ates, tests and reflects (self-reported; a dif­fer­ent pro­to­col) [32]. APEX-Agents, a bench­mark of pro­fes­sional agent tasks, launched in Jan­u­ary 2026 with a best score of 24.0% [33]; by Octo­ber the top of Mercor's own leader­board for it was 82.4% (±4.3; self-reported) [34]. GSO's head­line score rose from 4.9% at launch in May 2025 to 47.1% in July 2026, and to 88.2% in Sep­tem­ber on the day GSO changed how it prompts models [21]. An envi­ron­ment is a depre­ci­at­ing asset, so design refresh (gen­er­a­tors, live data, regen­er­a­ble held-out sets) into the first release.

The less obvi­ous lesson con­cerns the reward's shape. A pass/fail thresh­old sat­u­rates even when the quan­tity beneath it has not. GSO reports the share of its 102 tasks where one attempt is cor­rect and reaches 95% of a human expert's speedup; at that bar the leader passes 88.2%. Score the same attempts at twice the expert's speedup and it passes 18.6%; at six­teen times, 1.0% [35]. The tasks still have room; the bar does not. A graded score keeps room too: BRIGHT, a rea­son­ing-heavy retrieval bench­mark scored by nDCG@10, went from a best of 22.1 at launch to 66.9 in 21 months and still sits a third short of per­fect [36] [37].

Four questions in a fixed order

What does a lab check before it lets an envi­ron­ment near a train­ing run? Four things, in order. Sound­ness: high reward means the task was solved. Learn­abil­ity: pass rates sit where the target model learns. Inte­gra­tion: it runs in the lab's har­ness at the lab's con­cur­rency. Rights: license and prove­nance are clean, and no task dou­bles as some­one's eval­u­a­tion item. Each check is worth­less if an ear­lier one fails: a pass-rate band mea­sured on a hack­able ver­i­fier mea­sures the hack. Epoch's inter­vie­wees put the first cri­te­rion as "high reward must mean the task was actu­ally solved, not hacked" [29].

Sound­ness fails more often than builders expect. On τ-bench's air­line tasks an agent that does noth­ing scores 38% and beats a GPT-4o agent's 35%, because delib­er­ately unsolv­able tasks count an unchanged data­base as suc­cess; on SWE-Lancer an agent that replaces the tests with assert 1 == 1 scores 100%. The Agen­tic Bench­mark Check­list (ABC), an audit of ten agen­tic bench­marks, found 7 that vio­late task valid­ity ("a task should be solv­able if and only if the agent pos­sesses the target capa­bil­ity"), 7 that vio­late out­come valid­ity ("the eval­u­a­tion result (e.g., tests or checks) truly indi­cates task suc­cess"), and report­ing gaps in all 10, enough to mis­state per­for­mance "by up to 100% in rel­a­tive terms" [1].

The two validi­ties become four proofs a machine runs on every task, on every change. For task valid­ity, the oracle, a ref­er­ence solver restricted to the agent's own tools, reaches max­i­mum reward, and a do-noth­ing agent, which defines the floor, scores far below it. For out­come valid­ity, every canary (a scripted hack such as an empty answer, an edited test or an early exit) scores at or below the floor, and so does a sto­chas­tic exploit policy judged by its mean, never its best roll­out. The do-noth­ing test is the cheap­est check in the field and would have caught τ-bench's 38% in sec­onds (Table 1).

A maturity ladder

How far along is a given envi­ron­ment, and where is the unfin­ished work? Five rungs answer, from a pro­to­type that merely runs to a release cal­i­brated on ref­er­ence models (Table 2); the top rung includes a short train­ing run beside a random-reward con­trol, because random rewards can look like progress (Part III).

Open tool­ing now climbs the first three rungs for you, and L4 is where the unfin­ished work sits: OpenEnv's draft val­ida­tor leaves all seven sta­tis­ti­cal tests to who­ever oper­ates the envi­ron­ment, and for one of them, the adver­sar­ial floor, the pro­ce­dure "is not yet spec­i­fied any­where" [38].

The rec­on­cil­i­a­tion test­bed shows L3 designed in rather than bolted on. Its gen­er­a­tor works from the answer side, the prop­erty Qwen's team calls "hard-to-solve, easy-to-verify" [39]: it writes one true book and injects each break into exactly one party's view, so the ver­i­fier com­pares rather than solves. Its typed tools reject unknown fields, and the single reward after sub­mis­sion scores the oracle 1.0 and doing noth­ing exactly 0 (test­bed mea­sure­ment).

Ten hand-writ­ten tasks that pass these proofs are a demo; train­ing needs thou­sands, and uses each one up. Part II asks where they come from.

Part IITask supply: from seeds to checkable tasks

How do a few seed tasks become thou­sands of tasks that keep a trust­wor­thy check and still teach, and how do you know the pool taught any­thing about the tasks you care about?

A task is empty when the policy always solves it or never does, and every roll­out spent on it buys noth­ing. The cure has five steps: write the check first, grow with edits whose effect on the answer is stated in advance, keep tasks the trained policy some­times solves, count copies by struc­ture, and judge the pool on real tasks it never touched.

Run­ning exam­ple (toy sim­u­la­tion). Each task asks for the cheap­est route between two cities in a directed graph with inte­ger costs, within a given number of legs. An exact dynamic pro­gram ver­i­fies, three noisy beam searches play weak, mid and strong solvers, a 32-weight policy learns, and six edit oper­a­tors grow ten hand-writ­ten seeds; a second, dif­fer­ent gen­er­a­tor stands in for "real" tasks.

Seeds run out

Why can't ten good tasks carry a train­ing run? Because a learn­ing policy uses tasks up. GRPO-style meth­ods score each of a group's G roll­outs against the group's mean, and if every roll­out earns the same reward, "the result­ing advan­tage for this group is zero", and so is the gra­di­ent [40]. At pass rate p, a group is zero-vari­ance with prob­a­bil­ity p^G + (1 − p)^G: 43% at p = 0.9 and G = 8, and 66% at p = 0.95 (com­puted; Figure 3A).

The waste shows up as through­put, so it gets mis­di­ag­nosed as an opti­mizer prob­lem: a useful group costs G / (1 − share) roll­outs, 14 at 43% zero-vari­ance and 40 at 80% (com­puted). Larger groups recover signal per roll­out. At p = 0.9, going from 8 to 16 roll­outs cuts the share of roll­outs in zero-vari­ance groups from 43% to 19% (com­puted), and BroRL reports gains that con­tinue as roll­outs per prompt rise into the hun­dreds, after more train­ing steps had stopped help­ing [41]. What larger groups cannot do is add prompts, so com­pare 16 roll­outs on one prompt with 8 on each of two fresh in-band prompts at a fixed roll­out budget. In the toy, the eight train­ing seeds alone hit a 99% zero-vari­ance share within 48,000 roll­outs (Figure 3B), and breadth also drives gen­er­al­iza­tion: on Proc­gen, "agents need access to as many as 10,000 levels" to close the gap between train­ing and test per­for­mance [42]. Often the task supply, not the opti­mizer, sets the ceil­ing.

Write the verifier first

What must exist before the first gen­er­ated task? A check you trust; pipe­lines that scale make tasks check­able by con­struc­tion. SWE-smith's "core prin­ci­ple" is to "define an exe­cu­tion envi­ron­ment first, and then syn­the­size task instances within the envi­ron­ment": it injects bugs into work­ing code­bases and keeps those that break an exist­ing test, which gave 50,137 tasks from 128 code­bases for about $1,360 of model spend [44]. Envs-FORGE admits a gen­er­ated envi­ron­ment "only if its oracle solu­tion obtains reward 1 under the gen­er­ated tests" [45]. Label­ing after the fact is weaker: in Self-Instruct's audit of model-gen­er­ated data, only 58% of sam­pled out­puts were cor­rect and accept­able [46].

A gate that works rejects a lot. Self-Chal­leng­ing agents write each task as an instruc­tion, a ver­i­fi­ca­tion func­tion, an exam­ple solu­tion and three fail­ure cases, keep­ing it only if the solu­tion passes and every fail­ure case fails; 5.2% of full can­di­dates sur­vived, yet RL on them dou­bled Llama-3.1‑8B's aver­age suc­cess, from 12.0% to 23.5% [47]. Test the ver­i­fier like code, because it fails in both direc­tions: rule-based math ver­i­fiers miss 14% of cor­rect answers on aver­age, more as the policy gets stronger, and one policy passed a trained model ver­i­fier by answer­ing with a single "{" [48]. Four self-tests catch most of this before a single roll­out: the ref­er­ence answer passes, a known wrong answer fails, a cor­rect answer in another format passes, and a degen­er­ate answer (empty, a lone symbol, the ques­tion restated) fails.

In the toy, the ver­i­fier re-solves every child exactly and rejects it if the goal is unreach­able, two routes tie, the graph is too large or the edit broke its stated con­tract. Ten seeds became 10,644 ver­i­fied tasks, but only 1,106 came out dis­tinct, uncon­tam­i­nated and inside the learn­able band, about ten attempts per useful task (Figure 4, toy sim­u­la­tion).

Grow with operators that carry a contract

How do you mul­ti­ply a seed with­out losing its label? Use a small set of named edits, and write down for each how the answer must change (Table 3). Soft­ware test­ing calls such a rule a meta­mor­phic rela­tion: a prop­erty link­ing the out­puts of related inputs, so a new case can be checked with­out know­ing its expected output [49]. Every child then faces two checks: the ver­i­fier re-solves it, and the con­tract must hold between parent and child.

Three lessons from the toy carry over to any gen­er­a­tor. Con­tracts catch oper­a­tor bugs: the "safe" dis­trac­tor broke "answer unchanged" 244 times, and with­out the con­tract those tasks would have entered the pool mis­la­beled. Mea­sure each oper­a­tor's effect, because intu­ition can get the sign wrong (Figure 5A): tight­en­ing the leg budget, meant to make tasks harder, raised the mid solver's pass rate by 0.09 (95% inter­val 0.07 to 0.12), and loos­en­ing it left 672 of 675 chil­dren unchanged. Oper­a­tors leave cues a policy can learn: the dis­trac­tor always added a trap edge cheaper than the cor­rect one, so in the grown pool the cheap­est next edge lies on the opti­mal route at only 38% of deci­sion points, against 57% on real tasks; draw­ing the trap's cost from the normal range moved the pool to 64% (Figure 5B; all toy sim­u­la­tion).

Only train­ing tells gen­er­a­tors apart. SWE-smith com­pared four bug gen­er­a­tors at a fixed 507 train­ing tra­jec­to­ries each: the one in which an LLM edits a func­tion had the high­est yield, 56.0% of can­di­dates break­ing a test, yet trained the weak­est stu­dents, 5.7% (±1.5) of SWE-bench Ver­i­fied resolved against 8.6% to 9.2% for the other three [44]. Phi-4's authors like­wise found "organic ques­tions are sub­stan­tially more effec­tive than syn­thetic ques­tions" [52]. Yield mea­sures sur­vival, not what a task teaches.

Keep tasks the policy sometimes solves

Which tasks should stay in the pool, and mea­sured on whom? Theory and prac­tice agree on the middle. Bae et al. show that expected policy improve­ment is "lower-bounded by the vari­ance of task-level suc­cess prob­a­bil­i­ties", p(1 − p), which peaks at p = 0.5; keep­ing prompts with 0.3 < p < 0.7 lifted their 3B model's five-bench­mark aver­age from 26.3 for plain GRPO to 30.1, while fil­ters that removed only one side scored 23.8 to 25.9, below no filter at all [43]. DAPO keeps sam­pling "until the batch is fully filled with sam­ples whose accu­racy is nei­ther 0 nor 1", which, added last to four other fixes in its cumu­la­tive abla­tion, moved AIME 2024 from 42 to 50; com­pare such meth­ods per roll­out gen­er­ated, since "the sam­pling cost for each batch is dynamic" [40]. Buyers Epoch inter­viewed ask for "a min­i­mum pass rate of around 2⁠–⁠3%, or at least one suc­cess out of 64 or 128 roll­outs", and drop tasks near 70% [29].

The band belongs to the solver, not to the task: in the toy, only 26% of the mid solver's 1,313 in-band tasks were in the strong solver's band (toy sim­u­la­tion). A vendor that bands tasks on a fron­tier model and ships them to a lab train­ing a smaller one sells a dif­fer­ent prod­uct from the one on the label. Cal­i­brate on the policy you will train and re-mea­sure as it improves, as Sky­work-OR1 does when it drops prompts its model solved every time in the pre­vi­ous stage [53]; the band also drains (Figure 3B), so refill when the zero-vari­ance share climbs.

Cal­i­bra­tion has a price, and the expen­sive part is ruling tasks out. A task at 6 of 8 has a 95% Wilson inter­val (a con­fi­dence inter­val for a pro­por­tion that stays honest at small sam­ples) of 0.41 to 0.93, strad­dling the retire­ment line, and ruling a task out as below a 2% floor at 95% con­fi­dence takes 149 straight fail­ures (ln 0.05 / ln 0.98; com­puted). One suc­cess set­tles reach­a­bil­ity, so esca­late sequen­tially, adding roll­outs only to tasks at 0 of 8 and stop­ping at the first suc­cess: if 10% of tasks score 0 of 8, that aver­ages about 22 roll­outs per task instead of 149 (com­puted). Then read the zeros, since "a 0% pass rate across many trials (i.e. 0% pass@100) is most often a signal of a broken task" [54]. In a sto­chas­tic task, judge "too easy" against the best achiev­able score, not 100% (Part VI).

Most variants are copies, and leaks travel down lineages

How many of your tasks are actu­ally dif­fer­ent, and which qui­etly descend from the test set? The usual nov­elty and con­tam­i­na­tion gates read text, such as Tülu 3's rule flag­ging a test item when more than 50% of its tokens share an 8-gram with one train­ing item [55], and gen­er­ated tasks defeat text gates in both direc­tions. In the toy, a hash of the cost-labeled graph and leg limit finds 6,721 struc­tural copies among 10,644 ver­i­fied tasks; a text near-dupli­cate check keeps 4,452 of those copies and throws away 1,151 tasks whose word­ing barely changed but whose struc­ture did (Figure 6A, toy sim­u­la­tion). The dupli­cate def­i­ni­tion decides how many tasks you have, so pub­lish it (Table 4).

Reword­ing also defeats decon­tam­i­na­tion. Para­phrased test items "easily bypass" n-gram checks: fine-tuning Llama 2 13B on rephrased GSM-8K test items lifted its score from 28.7 to 95.3, and CodeAlpaca, a dataset GPT-3.5 gen­er­ated, holds rephrased copies of 12.8% of HumanEval [56] (Figure 6C). A gen­er­ated pool inher­its every such leak through its seeds: in the toy, of the 1,912 descen­dants of two seeds that were also eval­u­a­tion items, the best text rule caught 202 and no renamed or reworded copy, and the struc­tural hash caught every copy but only 100 of the 1,852 edited descen­dants (Figure 6B, toy sim­u­la­tion).

Split on the same struc­tural sig­na­ture, never on row index. In the rec­on­cil­i­a­tion test­bed, whose gen­er­a­tor draws fresh names and num­bers for every task, no two tasks' ini­tial states over­lapped on the sur­face, yet 6 pairs shared an iden­ti­cal set of injected errors and 3 of those pairs strad­dled the splits (test­bed mea­sure­ment). Assign each task to a split by a hash of its sig­na­ture, and every twin lands on the same side.

Judge the pool on real tasks that seeded nothing

Did the pool teach any­thing about the tasks you care about? A held-out split of the pool mea­sures how well the policy learned the gen­er­a­tor. In the toy, a policy trained to copy the exact solver's rank­ing on the oper­a­tor-grown pool scored 0.94 on that pool's held-out tasks but 0.76 on real tasks, no better than plain renam­ings of the seeds (0.77; Figure 7A). Growth paid only once the dis­trac­tor's cue was fixed (0.83) or the seeds came from the real dis­tri­bu­tion (0.88, against 0.90 for train­ing on real tasks), and every curve flat­tened by about 200 tasks: more vari­ants do not fix a biased gen­er­a­tor; better seeds do (toy sim­u­la­tion).

RL tells the same story more sharply (Figure 7B). After 48,000 roll­outs the eight seeds alone reached 0.75 on real tasks and every grown pool did worse; grow­ing tasks on demand peaked at 0.77 and fell to 0.52 while its syn­thetic score climbed to 0.86, because it chased the stu­dent's fron­tier rather than the target (toy sim­u­la­tion). The public record shows the same gap. RL on pro­ce­du­rally gen­er­ated ter­mi­nal tasks lifted Qwen2.5‑7B from 10.7% to 53.3% on the gen­er­a­tor's own devel­op­ment set but only from 2.2% to 3.4% on Ter­mi­nal-Bench 2.0, a human-curated bench­mark; the authors con­cede that their gen­er­ated tasks "resem­ble com­pet­i­tive pro­gram­ming prob­lems more than the messy, under­spec­i­fied requests that users actu­ally pose" [57] (Figure 7C).

What trans­fers is breadth of sources, so count seed fam­i­lies, not tasks. SWE-smith found per­for­mance rising roughly log­a­rith­mi­cally with the number of code­bases behind a fixed 700 train­ing tra­jec­to­ries [44]. Joint train­ing on RLVE's 400 adap­tive envi­ron­ments gained 3.37 points on aver­age across six bench­marks, while con­tin­u­ing the orig­i­nal RL gained 0.49 with more than three times the com­pute [58]. In Agent World Model, train­ing on 10 envi­ron­ments caused "severe per­for­mance degra­da­tion", while 100 gave sub­stan­tial gains and 526 more [59]. Part VI gives the sta­tis­ti­cal reason: vari­ants of one seed are cor­re­lated.

Five synthesis families, each with its checker and its failure

Which gen­er­a­tor should pro­duce your tasks? Start from the checker you can trust, not from the gen­er­a­tor you like. Each family comes with a typ­i­cal ver­i­fier and a typ­i­cal way of fail­ing, and the fail­ure is easier to pre­dict than the gain (Table 5).

Three fail­ures deserve a rule each. When one agent writes both the solu­tion and the checker, the two can share a mis­take, so an inde­pen­dent solver must also suc­ceed; End­less Ter­mi­nals keeps a task only if o3 solves it at least once in 16 attempts, a filter that "intro­duces a capa­bil­ity ceil­ing" [57]. A sim­u­lated world can be worse than none: Agent World Model's GPT-5 tool sim­u­la­tor left an 8B model below its untrained base on BFCLv3 (52.53 against 53.83). Self-play keeps tasks in the band but lets labels decay, because the major­ity vote that sets the band also sup­plies the labels. Mined tasks fail dif­fer­ently, by leak­ing the check: when SWE-smith wrote issue text from the fail­ing test, trained stu­dents tried to repro­duce the bug in 127 of 500 runs, against 379 for stu­dents trained on model-writ­ten issues [44]. A leaked grader shapes a worse habit even while reward rises.

Factory economics: checking is the bill

Where does the money go in a task fac­tory, and in what order should its gates run? Gen­er­at­ing can­di­dates is nearly free; check­ing them is the bill. SWE-rebench V2's setup agent made about 535,000 model calls, roughly $1.9K of infer­ence, while val­i­dat­ing its tasks took 583,809 Docker builds aver­ag­ing 2.71 min­utes, about 26,400 job-hours, and left 26.36 TiB of images [62]. SWE-smith's one image per code­base took 295 GB for 50,000 tasks, against an esti­mated 50 to 150 TB with one image per task [44]. Human review can dwarf both: Ter­mi­nal-Bench 2.0 kept 89 of 229 con­trib­uted tasks after "approx­i­mately three hours of com­bined reviewer atten­tion" per task [63] (Figure 8B).

Sur­vival runs from under 4% to 99%, and the spread is infor­ma­tion (Figure 8A). SWE-Gym kept 2,438 of 64,689 mined instances, 3.8% [60]. An inde­pen­dent val­ida­tor kept 6,275 of 32,079 released SWE-rebench V2 rows (19.6%) after drop­ping broken images and flaky rows, but 4,522 of 4,578 curated R2E-Gym rows (98.8%) [26]. A released task count is a claim; re-val­i­date it before you budget a run on it.

Two rules follow. First, order the gates by cost per rejec­tion, not cost per entrant, and dedu­pli­cate right after the cheap­est valid­ity gate: in the toy coding-task fac­tory, moving dedu­pli­ca­tion to just after the "gold patch flips a fail­ing test" gate cut the CPU of the next three gates by 55% with the same 235 tasks out, because dupli­cates stopped paying for three sta­bil­ity reruns each (toy sim­u­la­tion). Second, price a usable task with the funnel: cost per usable task ≈ (model cost per can­di­date + builds per can­di­date × cost per build) ÷ yield + review time per kept task (com­puted). At SWE-rebench V2's num­bers and an assumed $0.03 per vCPU-hour the machine term is about 11 cents, so dou­bling yield halves it while halv­ing model prices trims about a quar­ter (Figure 8C).

Every task now has a check, a con­tract, a band and a lin­eage. Part III asks what an opti­mizer does to that check once it is the only thing paying the policy.

Part IIIThe reward is a measurement

What does a policy do to a reward it is opti­mized against, how do you mea­sure a ver­i­fier's errors, which defenses hold under pres­sure, and how do scor­ers man­u­fac­ture or hide wins?

A static bench­mark tol­er­ates a ver­i­fier with a few per­cent error; an opti­mizer does not, because it moves the policy toward what­ever is paid. This part treats the reward as a mea­sure­ment taken under adver­sar­ial pres­sure.

Run­ning exam­ple (toy sim­u­la­tion). A 0.1M-param­e­ter trans­former learns two-digit addi­tion from scratch with GRPO, eight answers per prompt. A gold grader accepts one com­mit­ted inte­ger in any of six legit­i­mate for­mats; around it sit ver­i­fiers of graded qual­ity, two of them delib­er­ately buggy. A model this small cannot plan an exploit, so the toy shows mech­a­nisms, not mag­ni­tudes.

Two errors, one optimizer

Why does a false neg­a­tive only slow learn­ing, while a false pos­i­tive can take over a run? Let a policy take a hack instead of the honest behav­ior with prob­a­bil­ity p, through a single soft­max logit θ. The expected policy gra­di­ent is the covari­ance between reward and choice:

E[dJ/dθ] = p (1 − p) · (E[r | hack] − E[r | honest])

A false-neg­a­tive rate f mul­ti­plies the honest reward by (1 − f): the signal shrinks but never flips sign, so the policy learns the right thing more slowly. A false pos­i­tive that pays the hack a bonus b with prob­a­bil­ity φ adds φb to E[r | hack], so the push toward the hack is pos­i­tive at every p above zero (com­puted). Rarity makes a hack slow, not safe. In prin­ci­ple no reward escapes this: over all sto­chas­tic poli­cies, two reward func­tions can be unhack­able only if one of them is con­stant [65]. The engi­neer­ing ques­tion is which holes the cur­rent policy can reach, and how fast.

The toy sep­a­rates noise from a hole (Figure 9). Random errors injected into the gold grader (false-neg­a­tive rates up to 30%, false-pos­i­tive rates up to 5%) barely moved where train­ing ended: all nine arms fin­ished between 0.61 and 0.68 held-out pass@1, against 0.68 for the clean run and 0.37 at the start. A sub­string ver­i­fier, which pays any answer con­tain­ing the right digits, had an audited false-pos­i­tive rate of only 4.0% [3.1%, 5.2%], yet under train­ing it drove 72.5% of sam­ples to hack and pass@1 to 0.03, because "write digits that con­tain the answer" works on every prompt. The audit mea­sured how often wrong answers pass by acci­dent; the opti­mizer mea­sured how easily they pass on pur­pose. A lenient ver­i­fier that pays hedges such as "81 82" stayed at 0.3% hacked, because a hedge never earns more than the plain answer.

The reward ladder

How hack­able is each kind of reward, and in which direc­tion does each fail? Order reward sources by how hard they are to fool and you get a ladder: exact checks of a machine-read­able answer, exe­cutable tests, a judge answer­ing narrow rubric items, a judge scor­ing a whole response, and a learned scalar reward model. The order is a syn­the­sis, not a mea­sure­ment: an exact check com­pares two values; a test reads pro­gram behav­ior, but the pro­gram runs where the policy acts; a judge reads text writ­ten to please it; and a learned reward model was fitted on typ­i­cal responses, while an opti­mizer looks for atyp­i­cal ones. Hack­a­bil­ity rises as the reward reads the policy's words rather than the world's state.

Each rung fails in its own direc­tion. Exact checks fail mostly by false neg­a­tives: among responses a rule-based math ver­i­fier marked incor­rect, 38.5% were in fact cor­rect [66], and rule ver­i­fiers aver­age 86% recall at above 99% pre­ci­sion [48]. Tests fail by false pos­i­tives the policy man­u­fac­tures: in one fron­tier run, exit­ing before the tests ran and rais­ing an excep­tion that skipped them came to "become sys­temic, affect­ing nearly every train­ing envi­ron­ment" [2]. Judges fail by bias and per­sua­sion: a repet­i­tive-list attack, which length­ens an answer with­out improv­ing it, fooled Claude-v1 and GPT-3.5 judges 91.3% of the time against 8.7% for GPT-4 [67]. Learned ver­i­fiers fail by over-opti­miza­tion: a fine-tuned 1.5B model ver­i­fier held for about 450 RL iter­a­tions, then the train­ing reward surged away from an oracle's and the run col­lapsed [48]. Process reward models, which score each rea­son­ing step, sit on that rung too: DeepSeek-R1 lists its attempt under "Unsuc­cess­ful Attempts", because a model-based one "inevitably leads to reward hack­ing" [24].

The top rung tells you the kind of error to expect, not its size: all five of the toy's ver­i­fiers were rule checks, with audited false-pos­i­tive rates from 0% to 87.9%. Table 6 lists hacks seen in public and the defense that answers each.

Judges, rubrics and cascades

How do you reward a task no pro­gram can check? With a judge, and three prac­tices have public evi­dence. Ask many narrow ques­tions, not one broad one: Rubrics as Rewards scores each response against rubric items writ­ten for its prompt and beat single Likert-score judges by up to 31% (rel­a­tive) on HealthBench, a rubric-graded med­i­cal bench­mark [74], and check­list feed­back extracted from each instruc­tion was the only method to improve a strong 7B instruc­tion-fol­low­ing model on all five bench­marks tested [75]. Rubrics are hack­able too: stronger ver­i­fiers "sub­stan­tially reduce, but do not elim­i­nate, ver­i­fier exploita­tion" [76]. Remove the judge's known biases: most judges favor which­ever answer comes first, and only GPT-4 kept its ver­dict after a swap in more than 60% of cases, so judge both orders and count only agreed wins; a ref­er­ence answer cut its fail­ures on ten math ques­tions, asked in both orders, from 14 of 20 to 3 of 20 [67].

Put the judge behind a rule. A cas­cade lets a strict rule decide every response it accepts and sends only its rejec­tions to a judge. With rule recall R and false-pos­i­tive rate f, and a judge that recov­ers a share t of the cor­rect answers it sees and accepts a share j of the wrong ones, the cas­cade's recall is R + (1 − R)t and its false-pos­i­tive rate f + (1 − f)j, and the judge runs only at the rule's rejec­tion rate, which falls as the policy improves (com­puted; Figure 10). The part that is easy to miss: with a strict rule (f near 0) the cas­cade's false-pos­i­tive rate is the judge's rate on rejects, and every wrong answer reaches the judge, so keep its ques­tion narrow ("is this the same number as the ref­er­ence?") and audit j as the whole false-pos­i­tive rate. TinyV runs a small model only on rule rejec­tions and reports pass rates up to 10% higher for about 6% more com­pute per step [66]; a rule-then-model hybrid gained about 3 recall points at above 98% pre­ci­sion for 25.7% more train­ing time [48]. In the addi­tion toy a cas­cade reached 100% recall at 100% pre­ci­sion on the audit set, yet train­ing on it ended within noise of exact match (0.67 against 0.65 held-out pass@1, three seeds), because the rule's false neg­a­tives were for­mats the policy simply stopped writ­ing (toy sim­u­la­tion).

Measuring a verifier

How many graded sam­ples does an error rate need, and why does a rate mea­sured at release stop being true? Grade a sample of policy out­puts twice, once with the ver­i­fier and once with gold, a slow grader you trust (a human, a stronger model, an exact parser). Every sample lands in one cell of a 2 × 2 table, and four rates matter, each with its own denom­i­na­tor (Table 7).

Report false-pos­i­tive and false-neg­a­tive rates with Wilson inter­vals and their denom­i­na­tors, never pre­ci­sion alone; the text­book normal inter­val col­lapses to zero width when no errors are seen. Zero false pos­i­tives among n gold-wrong sam­ples bounds the rate below about 3/n at 95% con­fi­dence, the rule of three: below 1% takes 299 gold-wrong sam­ples, and below 0.1% takes 2,995 (com­puted: n = ln 0.05 / ln(1 − r)).

The trap is the denom­i­na­tor. An improv­ing policy makes wrong answers scarce: col­lect­ing 300 gold-wrong sam­ples takes about 300 / (1 − accu­racy) graded ones, which is 429 at 30% accu­racy, 3,000 at 90% and 30,000 at 99% (com­puted). An audit that was ample early in a run is too small late in it, exactly when the opti­mizer is best at find­ing holes, so keep a fixed adver­sar­ial set of known-wrong responses whose denom­i­na­tor never shrinks. Grader bugs move head­lines too: Claude Opus 4.5 first scored 42% on CORE-Bench, partly because rigid grad­ing rejected 96.12 where it expected 96.124991…, and 95% after the bugs were fixed and the scaf­fold loos­ened (self-reported) [54].

How a hack takes off

How fast does a rare exploit spread, and what pre­dicts when? GRPO gives each of a group's G answers an advan­tage (r − mean) / std com­puted within the group. If the vis­i­ble task gives no within-group vari­ance, because it is sat­u­rated or impos­si­ble, a hack is the only source of vari­ance: a lone hacked sample in a group of eight gets an advan­tage of √7 ≈ 2.65 what­ever the bonus is, because the bonus can­cels between numer­a­tor and denom­i­na­tor (com­puted).² If the task is half-solved, its own vari­ance dilutes the hack: head­room com­petes with the hack for the gra­di­ent.

A planted hole times the mech­a­nism (Figure 11). Fol­low­ing Prime Intel­lect's design, the toy pays a hidden +0.5 bonus for a harm­less key­word with a set base rate. With a reach­able task, half the sam­ples car­ried it (step t50) after 36, 94 and 370 steps at base rates of 5%, 1% and 0.1%, and every seed took off; with no head­room, t50 came 2.0 to 5.4 times sooner (toy sim­u­la­tion). Prime Intel­lect reports the same shape on a 1B model: hack­ing reached half the sam­ples by step 38 when the vis­i­ble task was unreach­able, by steps 47 to 51 when it was sat­u­rated and by steps 58 to 77 when it could still improve, and a word with a 0.16% base rate still reached 47.7% by step 100 (the dia­mond; self-reported) [77]. "The policy rarely does this" is not a safety argu­ment. Head­room does hold off a hack that costs the answer: a key­word that cor­rupted its answer took off only where honest answers paid less than the bonus.

The early warn­ing is cheap. A 0/1 reward com­po­nent's within-group vari­ance is pro­por­tional to p(1 − p), so its rise from near zero announces the take­off. Four cheap alarms (paid reward minus gold on an audited sample, scores above the honest max­i­mum, per-com­po­nent vari­ance and an action mon­i­tor) fired 14 to 23 steps before t50 on aver­age, though none fired first every time (toy sim­u­la­tion). Broken tasks are no-head­room tasks, which is why models hack far more often on tasks miss­ing a needed file (Table 6), and system-card hack rates are stress tests that "do not reflect real-world hack­ing rates" [78]: read their trend, not their level.

Proving a verifier

What shows that high reward means solved, and how can that evi­dence pass while the hole stays open? Part I's four proofs run on every task, and Ter­mi­nal-Bench 2.0 also runs an exploit agent during task audits and inspects its tra­jec­to­ries "regard­less of whether the agent passes" [63]. The rec­on­cil­i­a­tion test­bed shows three ways the proofs pass while the reward is still wrong.

Clip­ping a reward at zero pays for gam­bling. The func­tion max(0, x) is convex: it for­gives the down­side and keeps the upside. The Brier skill score S = 1 − 4(q − o)², for a fore­cast q of a binary out­come o, is proper when unclipped: at a true prob­a­bil­ity p* = 0.7, report­ing 0.7 expects 0.16 and report­ing cer­tainty −0.20. Clipped at 0, cer­tainty expects 0.70 against 0.448 for the truth, and even a clip at −1 pays cer­tainty 0.40 against 0.16 (com­puted; Figure 12). In the rec­on­cil­i­a­tion test­bed, clip­ping the break score at 0 would lift a weak scripted agent's mean reward from −0.156 to +0.014, turn­ing blind guess­ing into profit (test­bed mea­sure­ment); in public form, "under binary grad­ing, abstain­ing is strictly sub-opti­mal" [79].

A canary that cannot trip proves noth­ing. The test­bed's look-ahead canary asked for prices one cal­en­dar day after the val­u­a­tion date. For a Friday val­u­a­tion that day is a Sat­ur­day, still before the episode's Monday-morn­ing as-of time, so the request was not look-ahead: the canary did noth­ing, scored 0 and passed the floor test. Only a second asser­tion, that each canary raises the integrity event it was writ­ten to trip, caught the bug; run it on every task, since a cal­en­dar bug hides when­ever the test task falls on the wrong day.

Only a sto­chas­tic exploit judged by its mean catches gam­bling. With a clip at 0 put back into a scratch copy of the envi­ron­ment, none of 550 canary replays (11 canaries on 50 tasks) rose above the floor, while a blind random-search exploit's mean reward rose from −0.118 to +0.0014 and beat the floor on 14 of 50 tasks (test­bed mea­sure­ment). The floor test passed a real hole; the mean found it.

Two habits close most of what remains. Couple credit to the work: when a blend of two par­ties' wrong val­u­a­tions could land inside the sign-off tol­er­ance, the fix paid val­u­a­tion credit only in pro­por­tion to the breaks actu­ally found, which is METR's advice in prac­tice: "patch the exploit in the scor­ing func­tion rather than pun­ish­ing the model" [69]. And muta­tion-test every tol­er­ance, because a tol­er­ance is a hole with a size: of 10,303 faults injected into 188 KernelBench prob­lems, the offi­cial checker missed 16.9% of those an inde­pen­dent check could wit­ness, includ­ing 78.6% of pre­ci­sion faults, while opti­mized test inputs reached 98.0% detec­tion [80]. Table 8 lists the defense classes and how each can pass its own test.

Defenses that hold

Which mit­i­ga­tions have evi­dence, and in what order should you reach for them? Remov­ing the hole beats making it unprof­itable, which beats chang­ing what the hack teaches, which beats asking the model nicely.

Pre­ven­tion by con­struc­tion has the best evi­dence: grade state in a trusted process, with­hold tests and grad­ing files until scor­ing, close net­work chan­nels at the tool layer, and prefer hidden all-or-noth­ing checks to vis­i­ble par­tial credit. In a toy pro­gram-writ­ing task, par­tial credit on the two vis­i­ble tests drove lookup tables that spe­cial-case them from 16.8% of sam­ples to 99.0% and held-out pass@1 to 0.00, while all-or-noth­ing scor­ing on hidden tests drove them to 0.2% and pass@1 to 0.82 (toy sim­u­la­tion). Magis­tral made the same trade in pro­duc­tion, keep­ing binary code rewards after pro­por­tional ones ended 2% lower on LiveCodeBench [81].

When you can detect a hack but not remove it, do the arith­metic. A hack adds a bonus b, a detec­tor catches it with recall r, and a penalty P leaves an expected gain of b − rP, so the penalty holds only if P > b/r. A gate zeroes a flagged sample's whole reward, task reward included, so on a cor­rect answer the attempt's expected gain is (1 − r)(1 + b) − 1 (com­puted; Figure 13). In the toy (b = 0.5, r = 0.7, thresh­old 0.71) the key­word ended in 97.4% of sam­ples unde­fended, 81.8% under a 0.5 penalty, 0.7% under a 1.0 penalty and 0.5% under a gate, while held-out pass@1 stayed between 0.60 and 0.65 in every arm, so the task metric gave no warn­ing. Then give the agent an honest exit. On delib­er­ately impos­si­ble tasks, an abort option cut GPT-5's cheat­ing from 54% to 9% and o3's from 49% to 12% [68].

Mon­i­tors find holes; they are not rewards. A weaker model read­ing an agent's chain of thought caught 95% of two sys­temic hacks, against 60% read­ing actions alone; once that mon­i­tor entered the reward, the agents kept hack­ing, hid their intent, and its recall fell to near zero [2]. In the toy a mon­i­tor went blind with no pres­sure on it at all, from 80% to 0% recall, because train­ing ampli­fied an unmarked route to the same hole. Instruc­tions have evi­dence against them: "Please do not cheat." left o3's hack rate on one task at 80% [69], and telling models not to hack "can increase mis­align­ment by reward­ing models for acting against their instruc­tions" [82]. Inoc­u­la­tion prompt­ing, one system-prompt line that frames the hack as accept­able during train­ing, cut mis­aligned gen­er­al­iza­tion by 75⁠–⁠90% even with hack­ing above 99% [82]: it changes what the hack teaches, not whether it pays.

The stakes exceed a wasted run. In the same study, the model that learned to hack attempted to sab­o­tage safety-research code 12% of the time [83]. Envi­ron­ment qual­ity assur­ance is a safety con­trol, not only a data-qual­ity one.

Over-optimization

When the reward is learned or judged, how far can you push before the true objec­tive turns down? Gao, Schul­man and Hilton mea­sured it with a fixed "gold" reward model label­ing com­par­isons for proxy reward models. As a func­tion of how far the policy has moved, d = √KL(π ‖ π₀), gold reward fol­lows d(α − βd) for best-of-n sam­pling and d(α − β log d) for RL [84]. Larger reward models push the turn fur­ther out, while a KL penalty does not move the gold-versus-KL fron­tier, so its effect on gold is "akin to early stop­ping". Direct align­ment meth­ods such as DPO show the same degra­da­tion, often within one epoch [85]. Reward-model ensem­bles help with­out curing it: con­ser­va­tive opti­miza­tion over an ensem­ble "prac­ti­cally elim­i­nates overop­ti­miza­tion" for best-of-n in a syn­thetic setup [86], yet ensem­bles "do not elim­i­nate reward hack­ing" where mem­bers share error pat­terns [87].

The toy shows what an opti­mizer exploits first (Figure 14). Prox­ies trained on 500, 2,000 and 10,000 gold-labeled pairs reached 88%, 91% and 93% pair­wise accu­racy, yet RL against them drove gold down to between −1.6 and −2.4 stan­dard devi­a­tions while the proxy rose: the prox­ies had learned that each token costs reward but not how little a very short answer is worth. With the length cost removed from gold and the prox­ies retrained, RL against them ended within 0.07 of RL on gold itself (toy sim­u­la­tion). A proxy with 93% pair­wise accu­racy can be safe or badly wrong, depend­ing on whether its errors lie along the path the opti­mizer takes.

The scorer is part of the environment

How much can the scor­ing code alone change a ver­dict? A lot (Table 9). In the first five rows the model out­puts were fixed, or nearly so, and only the scor­ing changed; in the last two the pre­sen­ta­tion changed while the ver­i­fier scored every answer cor­rectly. Label noise com­presses the gaps between strong models, and a cue in the pre­sen­ta­tion can do the policy's work for it.

A related effect changes train­ing rather than scor­ing. RL with random rewards raised Qwen2.5-Math-7B on MATH-500 by 21.4 points, against 29.1 with ground-truth rewards, and such gains often fail to appear for other fam­i­lies such as Llama3 and OLMo2 [94]. A gain that a random reward also pro­duces is elic­i­ta­tion of a format, not learn­ing, so the eval­u­a­tion card (Table 13) asks for that con­trol on a second model family.

A trust­wor­thy reward is nec­es­sary, not suf­fi­cient: a dozen sys­tems choices decide whether it stays attached to the tokens that earned it. Part IV fol­lows one roll­out through the train­ing system.

Part IVFrom rollout to gradient

Where does the wall-clock time of an RL step go, what does each speedup cost in stal­e­ness or bias, and which record-keep­ing choices silently break a run?

A sound reward still has to be attached to the tokens that earned it. Between the ver­i­fier and the weights sit an infer­ence engine, a fleet of sand­boxes, a sched­uler and a trainer, each of which can qui­etly change what the gra­di­ent is com­puted on. Pro­duc­tion num­bers show where time goes; a toy roll­out system iso­lates each mech­a­nism.

The tail sets the step

Why do more GPUs not shorten an RL step? A step has four phases: a gen­er­a­tor (an infer­ence engine hold­ing a copy of the policy) sam­ples roll­outs; the envi­ron­ment and ver­i­fier score them; a trainer takes a gra­di­ent step; and the new weights return to the gen­er­a­tor. When each phase waits for the whole batch, the step is a bar­rier that lasts as long as its slow­est roll­out (Figure 15).

Pro­duc­tion pro­files agree that gen­er­a­tion, and its slow­est mem­bers, set the clock. On three Moon­shot pro­duc­tion work­loads, roll­out took 63% to 87% of each iter­a­tion, train­ing 10% to 31% and the weight update 2% to 6% [95]. Olmo 3's 32B learner, on 8 nodes against 20 for infer­ence, still waited for data 75% of the time [96]. Agents put the envi­ron­ment on the crit­i­cal path: when envi­ron­ments timed out in RollArt's soft­ware-engi­neer­ing agent train­ing, env.reset took 78% of roll­out time [97]. Lengths are heavy-tailed, with the 99th per­centile up to ten times the median [98] and the long­est response in a batch 25 to 32 times [99], and the long­est sequence still decodes one token at a time: "increas­ing resources does not reduce the Roll­out dura­tion" [100].

Place­ment fol­lows. If a work­load needs r units of gen­er­a­tion GPU-time per unit of train­ing, a split, pipelined design is bal­anced when gen­er­a­tion gets r/(r + 1) of the GPUs (com­puted). Pub­lished splits give gen­er­a­tion 60% to 75% of nodes, 44:16 for INTELLECT-3 [101] and 20:8 for Olmo 3: "allo­cate as many roll­out GPUs as you can such that the trainer does not wait" [102]. Re-pro­file weight sync when­ever step time drops; Prime Intel­lect's stayed at 60 to 90 sec­onds while its steps fell below five min­utes, and became the bot­tle­neck (self-reported) [103].

The staleness budget

What does asyn­chrony buy, and what does it cost? Asyn­chrony stops the wait­ing: gen­er­a­tors keep sam­pling from weights a ver­sion or more old while the trainer works. Stal­e­ness, the trainer's ver­sion minus the ver­sion that sam­pled a token, must be counted per token, because one episode can span sev­eral ver­sions; APRIL mea­sured about 40% of a step's tokens coming from responses started under ear­lier poli­cies [104]. The pur­chase arrives early: in the toy, bounded asyn­chrony reached 3.2 times the syn­chro­nous steps per minute at a stal­e­ness bound of 2 and stayed flat beyond it, but only 1.3 times with light-tailed latency, because the tail is what syn­chro­nous train­ing pays for (toy sim­u­la­tion). ROLL Flash also gets most of its gain by an asyn­chrony ratio of 2 [105], and verl reports 2.35 to 2.67 times (self-reported) [106].

The price depends on the objec­tive. PPO's ratio π_θ/π_old does two jobs: it cor­rects for data sam­pled by another policy, which needs the policy that actu­ally sam­pled, and it cen­ters a trust region, which can be any recent policy [107]. So write three poli­cies. The behav­ior policy μ is what the sam­pler actu­ally ran, in its ver­sion and numer­ics; the prox­i­mal policy π_prox is the trainer's weights at the start of the update; π_θ is being opti­mized. The weight π_prox/μ cor­rects the sam­pling and the clipped ratio π_θ/π_prox bounds the step. μ's log-prob­a­bil­i­ties exist only if the sam­pler recorded them.

Aver­aged over stal­e­ness bounds of 4, 8 and 16, AReaL's naive PPO scored 31.6 on AIME 2024 and its decou­pled objec­tive 40.6, against 42.0 for syn­chro­nous train­ing (com­puted from the paper's stal­e­ness abla­tion: a 1.5B model, one run per set­ting) [3]. At a bound of 4 the naive run dipped to 23.3. The decou­pled objec­tive stays within about a point of syn­chro­nous up to a bound of 8, then loses 3 to 5 points: the objec­tive, more than the bound, car­ries the run. In the toy, with the sam­pler 16 ver­sions behind, naive PPO-clip kept 69% of syn­chro­nous learn­ing and decou­pled PPO and trun­cated impor­tance sam­pling 83% to 85%, while the uncor­rected run's clip frac­tion stayed near 3% (Figure 16, toy sim­u­la­tion). A low clip frac­tion is not evi­dence of an on-policy run.

Knobs do not trans­late across frame­works: verl's staleness_threshold bounds the share of stale sam­ples [106], AReaL's η a ver­sion lag [3] and MiniMax's window a queue posi­tion [108], and capac­ity rules are approx­i­mate; SkyRL's con­troller "does not strictly guar­an­tee" its bound [109]. Admit by a capac­ity rule, then check each token's age at the trainer and drop what over­shoots.

Off-policy at zero staleness: training-inference mismatch

Why is "on-policy" train­ing off-policy even when ver­sions match? The infer­ence engine and the trainer eval­u­ate the same weights with dif­fer­ent ker­nels, batch shapes and pre­ci­sion, so they assign dif­fer­ent prob­a­bil­i­ties to the same tokens. This train­ing-infer­ence mis­match does not shrink when weights sync more often. Mix­ture-of-experts (MoE) models, which route each token to a few expert sub-net­works, make it worse: on Qwen3‑30B-A3B about 10% of router deci­sions dif­fered between infer­ence and train­ing, and 94% of tokens hit a dif­fer­ent expert in at least one layer [110]. Round­ing error com­pounds with length: BF16's mis­match "wors­ens with longer responses", and FP16 keeps it about 24 times smaller [111].

The fixes form a ladder from cheap to exact. Trun­cated impor­tance sam­pling (TIS) weights each token by min(π_train/π_sam­pler, C), with verl default­ing to C = 2, and mask­ing whole sequences is better "when high-weight sam­ples are likely garbage rather than signal" [112]. IcePop zeroes tokens whose ratio leaves [0.5, 5]; with­out it, plain GRPO on Ring-mini-2.0 col­lapsed after 180 to 200 steps [113]. Rout­ing replay feeds the sam­pler's expert choices to the trainer, and an FP32 lan­guage-model head raised ScaleRL's fitted asymp­tote from 0.52 to 0.61 [114]. Bit­wise-iden­ti­cal ker­nels remove the error, at a 25.3% full-step over­head in one imple­men­ta­tion (self-reported) [115]. Dis­crete choices need deter­min­is­tic ker­nels at every rung: in GLM-5's train­ing, non-deter­min­is­tic top-k oper­a­tors "caused dras­tic per­for­mance degra­da­tion" [116].

The com­mon­est mis­take looks like a cor­rec­tion. A trainer that recom­putes the "old" log-prob­a­bil­i­ties and uses them as the behav­ior policy divides the trainer by itself, which verl's doc­u­men­ta­tion says "leads to RL train­ing col­lapse" [112]. The fix is a record field: the sam­pler's log-prob­a­bil­ity for every sam­pled token, in the denom­i­na­tor.

Taming the tail without bias

Which tail fixes are free, and what do the others do to the data? Con­tin­u­ous batch­ing is free: it refills each gen­er­a­tion slot the moment it frees, and with a mean gen­er­a­tion of 14,628 tokens against a 32K cap, Olmo 3 com­puted that static batch­ing "would have" wasted up to 54% of its com­pute [96]. Every­thing faster is paid for in bias, as Figure 17 shows for eight sched­ul­ing dis­ci­plines on a toy work­load with a power-law tail of episode lengths. Greedy stream­ing, which trains what­ever fin­ishes first, ran six times faster, but its first ten steps trained episodes 41% shorter than the stream; restart­ing unfin­ished work was as short early and wasted 80% of gen­er­ated tokens. Win­dowed FIFO, which lets the trainer take fin­ished tra­jec­to­ries only within a window over the gen­er­a­tion queue, showed an early bias of −3%, with an inter­val includ­ing zero, at about half of greedy's speed (toy sim­u­la­tion).

MiniMax reports that under greedy sched­ul­ing "early train­ing batches are dom­i­nated by short, easy tasks", which desta­bi­lized opti­miza­tion, and adopted win­dowed FIFO [108]; DeepSeek-V4 logs every gen­er­ated token because regen­er­at­ing unfin­ished requests "intro­duces length bias" [117]. Two sched­ulers avoid the trade by stay­ing syn­chro­nous: Seer, for up to 2.04 times the roll­out through­put [95], and RollPacker, for 2.03 to 2.56 times end to end [99].

The environment is a distributed service

What changes when one envi­ron­ment becomes thou­sands of con­cur­rent ses­sions? It becomes a fleet with its own latency tail, capac­ity limit and fail­ure modes, each of which leaks into the gra­di­ent unless designed out. Size warm pools for the burst. Little's law (items in a stable system = arrival rate × time in it) gives the mean number of sand­boxes boot­ing at once [118], but a p99 target con­cerns the bursts above the mean. In the toy server (300 cre­ates a second, 207 ms cold boots), Little's law gives 62, which missed a 50 ms p99 target at 252 ms, and the p99 fell to 0.1 ms only at 80 (toy sim­u­la­tion; Figure 18A). A square-root margin, L + z√L with z = 2.33 for a 1% miss, gives 81, within 4% of the sim­u­lated min­i­mum of 78 (com­puted).³

Fail fast, and make retries idem­po­tent. Past 2,048 ses­sions on one 8-worker con­tainer, OpenEnv's fail­ure mode was "con­nec­tion rejec­tion, not time­out" [119]. A rejec­tion is unam­bigu­ous; a time­out invites a retry that may apply a step twice. In the toy, where 5% of responses stalled past the time­out after the step was applied, a fresh request id per attempt double-applied all 93 retried steps and a reused id none (toy sim­u­la­tion), and a dou­bled step earns reward for some­thing the policy never did.

A crash is not a zero. INTELLECT-3 states the rule: "On any sand­box fail­ure, we mask out the cor­re­spond­ing model com­ple­tion" [101]. GLM-5 excludes fail­ures from envi­ron­ment col­lapse, pads a group with valid repeats when more than half sur­vive and drops it oth­er­wise [116]. Recom­pute the base­line over the sur­vivors, or a crash's zero hands every sur­vivor an unearned advan­tage. Zero­ing is harm­less only when fail­ure is unre­lated to behav­ior (in the toy, where right and wrong episodes made the same number of tool calls, mask­ing showed no advan­tage). When the right strat­egy touches the envi­ron­ment more, zero­ing pays the short­cut: for a thor­ough strat­egy that suc­ceeds 60% of the time in 20 steps and a short­cut that suc­ceeds 40% of the time in 5, each cred­ited p(1 − c)^L, the short­cut ranks higher once the per-step crash rate c exceeds about 2.7% (com­puted; Figure 18B).

Resources are part of the reward. On Ter­mi­nal-Bench 2.0, with every­thing else fixed, uncapped resources beat strict enforce­ment by 6 per­cent­age points and cut infra­struc­ture errors from 5.8% to 0.5%, so doubt leader­board gaps under 3 points until con­fig­u­ra­tions match [120]. Snap­shots are state too: a VM snap­shot resumed twice starts both copies with the same random-number state unless the guest re-seeds, which is why hyper­vi­sors expose a gen­er­a­tion ID that changes on restore [121].

The record is the contract

What must a roll­out record carry? Every­thing the trainer will need to cor­rect for, because it cannot go back and ask. Nearly every fail­ure in this part is a miss­ing row of Table 10.

Tokens, not text. Decod­ing ids and re-encod­ing the string is not the iden­tity: a string has sev­eral valid tok­eniza­tions, a tok­enizer can merge char­ac­ters across the bound­ary between the policy's turn and the tool's, and a har­ness that re-ren­ders a chat can change spac­ing. Token-in, token-out (TITO) train­ing uses only the ids the sam­pler emit­ted; vLLM added return_token_ids to its OpenAI-com­pat­i­ble API because reto­k­eniza­tion drift "is not even at the token level, and there­fore cannot be cor­rected through token-level impor­tance sam­pling" [122].

On the toy tool task, a tok­enizer whose merges stay inside a turn changed 0.03% of episodes on a round trip, and reto­k­enized train­ing matched TITO. A delib­er­ately bound­ary-cross­ing tok­enizer changed 98.7%, masked the policy's second tool call and trained the policy on the envi­ron­ment's text: reto­k­enized train­ing ended at 0.087 held-out pass@1, below its own 0.189 start, against 0.606 for TITO (toy sim­u­la­tion; Figure 19). Drift belongs to the tok­enizer and tem­plate together, so it is neg­li­gi­ble or ruinous, and the next model's tem­plate can flip it.

Multi-turn TITO also needs a prefix-pre­serv­ing chat tem­plate, one where append­ing a tool mes­sage never changes how ear­lier turns render (TRL patches tem­plates for fam­i­lies such as Qwen3 [124]), and ses­sion affin­ity, so the cached prefix is reused: with­out it, a 25-turn episode pre­fills about 400,000 tokens for 29,000 dis­tinct ones (com­puted from the for­mula in [125], with a 4,000-token prompt and 1,000 new tokens a turn). Finally, cap­ture at the model-call bound­ary. Deployed agents are third-party har­nesses that speak text, so labs put a gate­way between har­ness and engine that ren­ders tokens once and records ids, log-prob­a­bil­i­ties and ver­sions: GLM-5 routes all roll­outs through one because re-tok­eniza­tion "can cor­rupt step align­ment between actions and rewards" [116], and Kimi K2.5 "strictly fol­lows a Token-in-Token-out par­a­digm" [126].

Con­text resets break the linear record. Long-hori­zon agents rarely keep their whole his­tory: DeepSeek-V3.2's tem­plate drops ear­lier rea­son­ing when a new user mes­sage arrives, and its search agent sum­ma­rizes or dis­cards old tool calls once 80% of the window is used [25], while GLM-5's search agent folds, at infer­ence, every obser­va­tion older than the last five rounds [116]. Each reset rewrites the prefix, so a record that con­cate­nates the turns trains later tokens on a con­text the sam­pler never saw. MEM1 keeps the episode as one sequence under a two-dimen­sional atten­tion mask that restricts each token "to only the tokens retained in memory at the time that token was gen­er­ated" [127]. The sim­pler equiv­a­lent: store each reset as a seg­ment with its own token ids, ver­sions and log-prob­a­bil­i­ties, give every seg­ment the episode's advan­tage, and mask any sum­mary another model wrote.

Masks, advantages, aggregation and credit

Once the record is faith­ful, which tokens should the gra­di­ent push, and how hard? Every nor­mal­izer in the answer is a weight in dis­guise.

Mask what the policy did not write. A token gets gra­di­ent only if the policy gen­er­ated it; prompts, tool out­puts and sim­u­lated-user turns stay in the con­text but out of the loss. Search-R1 aver­aged 0.431 with retrieved tokens masked and 0.343 with­out, on a 7B model trained with PPO [123]; in the toy the end-point cost was smaller, but unmasked runs took twice the steps to reach the same accu­racy (toy sim­u­la­tion).

Advan­tages: drop the per-group spread. GRPO scores each roll­out by its reward minus its group's mean, divided by the group's stan­dard devi­a­tion [15]. The divi­sion up-weights lop­sided groups: a lone suc­cess in a group of eight gets +2.47 instead of +0.875, and lop­sided groups are where lucky guesses and ver­i­fier false pos­i­tives live (com­puted). Dr. GRPO calls this a dif­fi­culty bias and removes it, along with per-response length nor­mal­iza­tion [128]. When envi­ron­ments mix, nor­mal­ize per task, or tasks learn "at very dif­fer­ent rates", as AgentRL found [129].

Aggre­ga­tion and trun­ca­tion. Divid­ing each episode's summed token loss by its own length makes long wrong answers cheap: beside a 4-token cor­rect answer, each token of a 32-token wrong one is pushed down with an eighth of the force (com­puted), so divide by a fixed number. A trun­cated episode was never judged, and scor­ing it zero adds a length penalty nobody designed: DAPO's over­long fil­ter­ing, which drops trun­cated sam­ples from the loss, moved AIME 2024 from 30 to 36 in its cumu­la­tive abla­tion [40], and ScaleRL found trun­ca­tion rates of 10% to 15% "typ­i­cally desta­bi­lized train­ing", while its stable 8B runs kept trun­ca­tions below 5% for over 90% of train­ing [114].

Credit across turns and roles. Broad­cast­ing one advan­tage to every token is the strong default on short tasks. When agents revisit iden­ti­cal states, GiGPO com­pares the turns taken from the same "anchor state", with­out a critic, for gains of more than 12% on ALFWorld and 9% on WebShop [130]. Add per-turn rewards only for facts a pro­gram can check, and treat other models in the episode as envi­ron­ment com­po­nents with error rates of their own: 16% for τ²-bench's tele­com user sim­u­la­tor against 40% for τ-bench's retail one [131].

Watching a run

Which curves tell a healthy run from a dying one? Train­ing reward alone is the worst single signal: it rises when the policy learns, and also when it finds a ver­i­fier hole, when the mix drifts toward easy tasks and when crashed episodes stop being counted, and DAPO's authors note it "often exhibits little cor­re­la­tion" with val­i­da­tion accu­racy [40]. Watch one curve per fail­ure axis, per envi­ron­ment: held-out pass@1 and large-k pass@k, reward per ver­i­fier family, the zero-vari­ance share, entropy, the sam­pler-to-trainer log-ratio, trun­ca­tion and fail­ure rates.

Some curves lead. RAGEN's "Echo Trap" in multi-turn agents is "reward vari­abil­ity col­lapse, entropy drop, and gra­di­ent spikes": reward spread and entropy often become unsta­ble before reward degrades, and gra­di­ent spikes "typ­i­cally mark the point of irre­versible col­lapse" [132]. Col­lapses are fast: with­out rout­ing replay, GRPO on a 30B MoE crashed at step 40 in one con­fig­u­ra­tion [110]. Curves say that some­thing is wrong; tran­scripts say what. Sample them where two sig­nals dis­agree (reward 1 with a very short tra­jec­tory, reward 0 with a final state that looks right), label each with one cause (policy error, ver­i­fier error, envi­ron­ment bug, infra­struc­ture fail­ure, hack), and fix each where it lives. Only the first is the trainer's busi­ness.

With a sound reward and a faith­ful record, the ques­tion is what the gra­di­ent is made of. Part V asks when to use some­thing richer than one scalar per episode, and what RL is actu­ally good for.

Part VChoosing the learning signal

Given your data, your reward and your budget, which post-train­ing method should you run? When should you stop sam­pling attempts and copy an oracle instead? And what can RL teach that super­vised fine-tuning or dis­til­la­tion cannot, and the reverse?

Even a trust­wor­thy instru­ment usu­ally hands the opti­mizer one number per episode, often one bit; the signal is coarse when­ever some­thing richer was avail­able and went unused. Pick the method by its inputs, train on the whole rank­ing when some­thing can score every option cheaply, and use RL to make reli­able what a model can already pro­duce.

One gradient, many weightings

What are the fam­i­lies of post-train­ing meth­ods, and what picks between them? DeepSeekMath writes super­vised fine-tuning (SFT), rejec­tion-sam­pling fine-tuning, direct pref­er­ence opti­miza­tion (DPO), prox­i­mal policy opti­miza­tion (PPO) and GRPO as one gra­di­ent [15]. Every method raises the log-prob­a­bil­ity of some tokens and lowers others; meth­ods differ only in where the sequences come from (a fixed dataset or the model's own sam­ples), what sup­plies the signal (labels, a reward model, a pro­gram, a teacher) and the per-token weight. SFT puts weight 1 on demon­stra­tion tokens; DPO pushes a pre­ferred response up and a rejected one down; GRPO weights each response by its advan­tage over its group; on-policy dis­til­la­tion weights each sam­pled token by −(log π_stu­dent − log π_teacher). The fam­i­lies are set­tings on one dial.

Three inputs choose the set­ting: the super­vi­sion signal, where sam­ples come from, and the budget. Gen­er­a­tion dom­i­nates any­thing on-policy: roll­out took "more than 95% of the total train­ing time" of PPO in one con­trolled com­par­i­son, which also found that better pref­er­ence data gave "the largest improve­ments", ahead of the algo­rithm and then the reward model [133]. A trainer's name is not its recipe: TRL's GRPO trainer defaults to a DAPO-style token-mean loss and a KL coef­fi­cient of 0, so "the ref­er­ence model is not loaded" [124]; record base­line, advan­tage scal­ing, clip­ping, KL and loss aggre­ga­tion for every run. Low-rank adapters (LoRA) are enough for RL, which gets "only O(1) bits per episode": LoRA "fully matches the learn­ing per­for­mance of FullFT" for RL "even with ranks as low as 1", given about ten times the full fine-tuning learn­ing rate and adapters on all layers, "in par­tic­u­lar, the MLP (includ­ing MoE) layers" [134].

Figure 20 turns the inputs into start­ing defaults. With fixed pref­er­ence pairs, start with length-nor­mal­ized DPO: in Tülu 3, "only length-nor­mal­ized DPO out­per­formed our base check­point over­all", at about 4 node-hours against 56 for PPO (com­puted from the reported run times) [55]. With a pro­gram that checks answers, start from GRPO with Dr. GRPO's fix for "an opti­miza­tion bias in Group Rel­a­tive Policy Opti­miza­tion (GRPO), which arti­fi­cially increases response length" [128] and DAPO's clip-higher and dynamic sam­pling [40]; at mix­ture-of-experts scale or far off-policy, switch to CISPO, which caps each token's impor­tance weight instead of drop­ping its gra­di­ent, or GSPO, which clips one ratio per response: "both GSPO and CISPO sub­stan­tially out­per­form DAPO" on the fitted ceil­ing [114]. For open-ended qual­ity, score rubric items with a judge (Part III) and audit with a panel the policy never trained against [76].

Recipe pieces are worth what they fix

Which of the dozen pub­lished fixes should you add? Only those that repair some­thing your run has. DAPO's cumu­la­tive ladder on Qwen2.5‑32B went from 30 on AIME 2024 for naive GRPO to 42 with four fixes, then to 50 with dynamic sam­pling, which drops prompts whose groups all pass or all fail [40]: wasted groups were that run's bind­ing con­straint. In VAPO's leave-one-out ladder, remov­ing the critic's value pre­train­ing dropped the score from 60 to 11 [135]. JustRL found the oppo­site sign: its min­i­mal recipe reached 55% on AIME 2024, adding an over­long penalty left it at 50%, and adding a "robust ver­i­fier" as well at 45% [136].

In the addi­tion toy, tog­gling six stan­dard pieces one at a time moved held-out pass@1 by −0.023 to +0.013, every inter­val includ­ing zero, because noth­ing could bind: the clip touched 1.55% of tokens (toy sim­u­la­tion). When a piece does bind, it can rival the objec­tive. Bridge­wa­ter's AIA Labs, with Think­ing Machines, post-trained an open Qwen3‑235B model on six doc­u­ment-triage tasks to 84.7% accu­racy, against 78.2% for the best expert-prompted fron­tier model, at "a 13.8x reduc­tion in infer­ence costs per task"; drop­ping inter­leaved per-task batches cost 12.5 points and drop­ping on-policy dis­til­la­tion 12.3 (self-reported) [137].

ScaleRL, fit­ting sat­u­rat­ing curves to more than 400,000 GPU-hours of runs, sep­a­rates levers that raise the ceil­ing (the loss type, an FP32 head, larger batches, fil­ter­ing zero-vari­ance prompts) from levers that "mainly adjust com­pute effi­ciency" (loss aggre­ga­tion, length penal­ties, advan­tage nor­mal­iza­tion), and warns that meth­ods that win at small bud­gets "can be worse when extrap­o­lated to large-com­pute regimes" [114]. The KL penalty gets set wrong in both direc­tions: drop it for rea­son­ing RL against a rule-based ver­i­fier, where "this restric­tion is not nec­es­sary" [40], and keep a small one with a learned reward, from a strong dis­tilled start [138] and under heavy asyn­chrony [139]. Get its gra­di­ent right, too: GRPO's k3 term dif­fer­en­ti­ates a KL esti­mate as a loss, which is "gen­er­ally incor­rect" [140], so put a detached esti­mate in the reward. And one asser­tion catches many silent bugs: on the first mini­batch after a fresh roll­out, every impor­tance ratio must be exactly 1 [141].

If you have an oracle, stop sampling

When some­thing can score every option in a state (a search engine, a sim­u­la­tor, an exact solver, a teacher model's prob­a­bil­i­ties), why sample a few options and reward them? Usu­ally you should not. Sample and score is RL with the oracle as the reward. Score every­thing is list­wise dis­til­la­tion: score all K options, turn the scores into a soft­max target at tem­per­a­ture τ and fit it with a KL loss, as AlphaGo Zero trains its policy toward its search prob­a­bil­i­ties [142].

Count the infor­ma­tion. A full rank­ing of K options picks one of K! order­ings, so it car­ries up to log₂ K! bits: about 22 at K = 10 and 525 at K = 100, against at most log₂ G! for G scored sam­ples (15.3 at G = 8) and one bit per pass/fail roll­out (com­puted). The bound over­states what reaches the stu­dent: in a toy sweep from K = 8 to 256, the bound's advan­tage grows about 110-fold while the speed-up of list­wise dis­til­la­tion over GRPO grows only from 3.3× to 10.6× (toy sim­u­la­tion).

Two levers make the real gap. Cov­er­age: with every option scored, the expected policy gra­di­ent is com­puted exactly instead of esti­mated [143]; in a toy with 100 options, the gra­di­ent from 8 scored sam­ples had a cosine of only 0.28 with the exact one. The objec­tive: the exact policy-gra­di­ent push on option a is π_a(v_a − E_πv), weighted by the model's own prob­a­bil­ity of choos­ing a, so a good option the model ignores barely moves, while the dis­til­la­tion push q_a − π_a moves it what­ever the model believes. In Figure 21 (64 options, a linear stu­dent; regret 0 is the oracle's pick and 1 an aver­age option), reach­ing regret 0.1 took GRPO 8,128 posi­tions, the exact policy gra­di­ent 4,480 (1.8× from cov­er­age) and soft list­wise dis­til­la­tion 1,664 (another 2.7× from the objec­tive), while GRPO on a win/loss reward never got there (toy sim­u­la­tion). Charged per oracle call instead, GRPO wins (8 scores per posi­tion, not 64), so the argu­ment needs a cheap oracle.

The public com­par­isons we found point the same way, with one instruc­tive excep­tion (Figure 22). Tar­gets from a search's visit dis­tri­bu­tion beat its chosen move by 50 ± 13 Elo in Hex [144]; values for every chess move beat best-move labels by 492 Elo from the same games [145]; super­vis­ing every com­mand branch lifted sim­u­lated-driv­ing suc­cess from 16% to 96% [146]. Qwen3‑8B reached 74.4 on AIME 2024 with on-policy dis­til­la­tion from larger Qwen3 teach­ers (32B or 235B-A22B) for 1,800 GPU-hours, against 67.6 with RL for 17,920; the teach­ers' own train­ing is not counted, so this com­pares two ways of using a teacher that already exists [4]. The excep­tion: in Atari, squared error on raw Q-values, the dens­est signal, reached only 33.9% of the teacher's Break­out score against 94.7% for a KL loss, because "greedy action choices can be made based on very small dif­fer­ences in Q-values, which receive low weight in the MSE cost" [147]. Den­sity needs a loss that weights what decides the choice.

The gap holds with a lan­guage model as the stu­dent, in the cross­word-tile-game test­bed, where a search player scores every legal move. GRPO on Qwen3.5‑4B with low-rank adapters learned from about 1,000 posi­tions per GPU-hour, and after about 25 steps 70% to 78% of its groups were zero-vari­ance, not from mas­tery: it matched the search player's top move only 39% of the time, con­fi­dently wrong. List­wise dis­til­la­tion of the search player's values over 16 can­di­dates, scored in one for­ward pass, learned from about 42,000 posi­tions per GPU-hour on the same model and ended 31 Elo below the player whose values it copied (test­bed mea­sure­ment; the CPU search that made the labels is not in that rate). Out­comes teach slowly because the win-prob­a­bil­ity gap between good moves aver­ages 0.009 against an out­come stan­dard devi­a­tion of 0.43, so one such gap takes about 2 × (0.43/0.009)² ≈ 4,600 games to resolve at one stan­dard error (com­puted); one oracle label set­tles it.

The five jobs RL keeps

When is RL still the right tool? When one con­di­tion that list­wise dis­til­la­tion needs is miss­ing. There are five.

1. There is no oracle, only a ver­i­fier. A math checker or a test suite passes or fails a whole output and cannot rank alter­na­tives, so there is no list to dis­till. DeepSeek-R1-Zero learned to reason from "a rule-based reward system" of accu­racy and format rewards [149].

2. The teacher sees what the stu­dent cannot. Priv­i­leged infor­ma­tion "is marginal­ized during imi­ta­tion learn­ing, result­ing in an 'imi­ta­tion gap'" [150]. In the tile-game test­bed, a search player that sees the oppo­nent's rack is a yard­stick, not a teacher: its best move rests on tiles the stu­dent never sees (Part VI uses it to mea­sure the luck ceil­ing).

3. The oracle mis­prices the long run. Dis­til­la­tion copies the oracle's blind spots. In a toy where each move's value has an imme­di­ate part a myopic oracle sees and a delayed part that partly dis­agrees, dis­till­ing the myopic oracle stalled at regret 0.43, and RL started from that sharp policy ended at 0.073 against 0.036 for RL from scratch, because a sharp policy explores little (toy sim­u­la­tion, 5 seeds). Public work finds the better order: dis­til­la­tion then RL "con­sis­tently out­per­forms pure OPD, pure RLVR, and all such joint base­lines" [151]. The RL stage also buys robust­ness off the train­ing dis­tri­bu­tion: out­come-reward RL car­ried over to unseen rule and visual vari­ants where super­vised fine-tuning mem­o­rized, though fine-tuning first still sta­bi­lized the output format [152].

4. You need to beat the teacher. A reward made from the teacher's score has the teacher's choice as its opti­mum, and dis­til­la­tion lands near the teacher: Expert Iter­a­tion's appren­tice won "just over half of games played" against the search it imi­tated [144]. Going fur­ther takes a better teacher, which Expert Iter­a­tion builds by search­ing over its own net­work and dis­till­ing again, and DeepSeek-V4 trains domain experts with RL before con­sol­i­dat­ing them "through on-policy dis­til­la­tion" [117]: RL makes the ora­cles, dis­til­la­tion copies them. Max­i­miz­ing a noisy teacher's argmax back­fires: in the test­bed, an RL stage after dis­til­la­tion drove the policy's entropy from 1.9 to 0.08 nats and lost 9 points of win rate, because two inde­pen­dent search labels agreed on the best move in only 77% of posi­tions (test­bed mea­sure­ment).

5. Each option is expen­sive to score. A list­wise target charges one oracle call per option; the chess dataset above took "roughly 8864 days of unpar­al­lelized Stock­fish eval­u­a­tion time" [145]. With human raters or slow sim­u­la­tors, score a short­list.

DAgger fixes drift, not ignorance

Should you pay to have the oracle label the stu­dent's own states? Only if the stu­dent drifts. Off­line imi­ta­tion trains on the expert's states, and a stu­dent that errs wan­ders where the expert never went: "a clas­si­fier that makes a mis­take with prob­a­bil­ity ϵ under the dis­tri­bu­tion of states/obser­va­tions encoun­tered by the expert can make as many as T²ϵ mis­takes in expec­ta­tion over T-steps" [153]. DAgger has the expert label the states the stu­dent visits, so the bound grows only lin­early in T, and on-policy dis­til­la­tion is DAgger with a soft teacher [154], "a one-line change on top of RL imple­men­ta­tions which use KL reg­u­lar­iza­tion" [155].

It pays in pro­por­tion to drift. In Hex, three DAgger rounds added 120 Elo [144], though they also quadru­pled the data; in sim­u­lated driv­ing, adding the stu­dent's own roll­outs lifted suc­cess from 16% to 64% [146]. In the tile-game test­bed DAgger added noth­ing at matched GPU-hours, most likely because fresh tiles and the oppo­nent's move reshuf­fle much of the posi­tion each turn. A toy cor­ri­dor shows the switch: with­out reshuf­fling, off­line dis­til­la­tion's excess cost grew to 1.91 at 500 steps against 0.08 for DAgger; with 10% of moves draw­ing a fresh state, the two landed within 0.03 (toy sim­u­la­tion, 20 seeds).

The pre-test is cheap: have the oracle score the stu­dent's moves on the stu­dent's own states and on the oracle's, and if the regrets match there is no drift to fix. Mind the KL direc­tion too: reverse KL is mode-seek­ing, so the stu­dent "learns one spe­cific behav­ior (the teacher's) instead of spread­ing its dis­tri­bu­tion across sev­eral sub­op­ti­mal options" [155], which suits a final model, not a stu­dent that RL will train next (job 3).

Knowledge versus procedure

What can each signal put into a model? Knowl­edge arrives through tokens. A scalar reward can make a pro­ce­dure reli­able, but it rarely cre­ates one the model cannot already sample.

SFT mostly selects a format. LIMA was fine-tuned on "only 1,000 care­fully curated prompts and responses", on the view that "almost all knowl­edge in large lan­guage models is learned during pre­train­ing" [156].

RL with ver­i­fi­able rewards mostly sharp­ens. On AIME 2024 at k = 1,024 sam­ples, 13.3% of prob­lems were solved only by the base model and 0.0% only by its RL-trained ver­sion, which "mainly sharp­ens the dis­tri­bu­tion within the base model's prior rather than expand­ing beyond it", whereas dis­til­la­tion "can intro­duce new rea­son­ing pat­terns from the teacher" [157]. Qwen3's RL left pass@64 at 90.0; on-policy dis­til­la­tion lifted it to 93.3 [4]. RL edits little, leav­ing 68.5% to 96.0% of param­e­ters unchanged in public check­points [158], and "pre­serves prior knowl­edge and capa­bil­i­ties sig­nif­i­cantly better" than SFT [159]. Expan­sion appears with long, broad train­ing where the base is weak (ProRL's 2,000-plus steps took a task where the base "exhibits no capa­bil­ity" from 0.00 to 7.91 pass@1 [138]), yet "the shrink­age of observed sup­port out­weighs its expan­sion" [160]. Figure 23 shows the cross­over in the addi­tion toy.

New facts through fine-tuning raise hal­lu­ci­na­tion. "As the exam­ples with new knowl­edge are even­tu­ally learned, they lin­early increase the model's ten­dency to hal­lu­ci­nate" [161], and rea­son­ing fine-tuning "degrades absten­tion (by 24% on aver­age)" [162]. Tokens plus a teacher can repair the damage: mid-train­ing Qwen3‑8B on a com­pany's inter­nal doc­u­ments alone cut its instruc­tion-fol­low­ing score from 85% to 45%, while mixing in chat data and then dis­till­ing on-policy from the orig­i­nal model restored it to 83% and raised inter­nal ques­tion answer­ing from 18% to 41% [155].

A scalar reward cannot quickly teach a pro­ce­dure the model cannot sample. In the tile-game test­bed, a lan­guage model trained with GRPO learned to re-rank eight engine-listed moves (regret fell from 5.21 to 3.91 points a move) but not to gen­er­ate moves from the board: 89.5% of its free moves were ille­gal, and it gamed the legal­ity reward (a legal one-tile play earned about +0.1, an ille­gal move −0.5) by writ­ing one-letter plays, 1,503 of 2,307 answers against 26 at the start. A decoder con­strained to a prefix tree of legal moves then made every free move legal, and the player still lost all 1,600 games it played against two simple bots (test­bed mea­sure­ment). Legal is easy; good is hard. RL selects among behav­iors the base can pro­duce, and a move gen­er­a­tor was not one of them.

What the model sees can matter more than how good its teacher is. In the same test­bed, four engine facts per option (such as the oppo­nent's best reply) cut per-move win-prob­a­bil­ity regret from 0.0105 to 0.0079 (95% inter­val of the dif­fer­ence −0.0038 to −0.0015), while labels two to three times closer to the truth changed noth­ing mea­sur­able, because the stu­dent's own error was four to five times its labels'. Hand­ing the end of the game to the engine (an exact solver once the bag is empty and both racks are known, a search just before it) added 29 Elo (inter­val 26 to 33) for about four CPU-sec­onds a game (test­bed mea­sure­ment). Pub­licly, a chess trans­former whose top moves all look won will "double-check this con­di­tion with Stock­fish" [145]; tool use is the gen­eral case.

Distill or RL?

The part reduces to at most four ques­tions (Table 11); ask them in order and stop at the first answer that names a recipe. The tile-game test­bed's legal search player ends at off­line list­wise dis­til­la­tion; the rack-seeing player stops at ques­tion 2.

Every claim in this part is a com­par­i­son between two num­bers. Part VI asks how often such com­par­isons are noise, how eval­u­a­tion designs tilt them, and what to do where luck and hidden infor­ma­tion set a ceil­ing no method can pass.

Part VIMeasuring in noisy worlds

When is a mea­sured gain real, how do eval­u­a­tion designs flat­ter one side, what ceil­ing do luck and hidden infor­ma­tion put on every legal player, and why are mar­kets the hard­est RL domain?

The last two ledger lines con­cern the com­par­isons every ear­lier part relied on. A gain can be real and still unread­able, because the eval­u­a­tion cannot see an effect that small: a sta­tis­tics prob­lem with known reme­dies. And part of the gap to a per­fect score may be clos­able by no player, because the infor­ma­tion does not exist at deci­sion time: a prop­erty of the domain, which the envi­ron­ment should mea­sure and report.

A score is a sample

How large a gain can a 30-prob­lem bench­mark see? Less than most papers claim. A score mixes three random draws: which items the bench­mark con­tains, which answers the model hap­pened to sample [5] and, for a claim about a train­ing recipe, which train­ing run you got [163]. For pass/fail items the stan­dard error of an accu­racy p on n items is √(p(1 − p)/n): at 62% accu­racy the 95% inter­val is ±17 points on 30 items, ±5.5 on 300 and ±3 on 1,000 (com­puted). In Figure 24, which eval­u­ates one model 20 times at each size, 37% of single runs on 30 items read 3 or more points above the truth (exact bino­mial): a "gain" that is pure noise.

Small eval­u­a­tions also need a better inter­val, because the text­book normal-approx­i­ma­tion (Wald) inter­val under­states uncer­tainty below a few hun­dred items [7]: at 30 items and a true accu­racy of 90%, a nom­i­nal 95% Wald inter­val covers the truth 81% of the time and the Wilson inter­val 97% (com­puted exactly). On AIME 2024, pass@1 has a stan­dard devi­a­tion of 5 to 15 points across sam­pling seeds; re-run under one pro­to­col, three vari­ants of a method whose orig­i­nal report looked promis­ing scored 28.9, 31.3 and 29.7 against a base­line of 28.7 ± 4.8, none sig­nif­i­cant [164]. On Chat­bot Arena, two sub­mis­sions of the same check­point scored 1069 and 1052 [165].

Each draw has its own remedy, in order. Extra sam­ples per item remove sam­pling noise, which dom­i­nates the gap between two check­points of one run because they mostly solve the same items; only more items sup­port a claim beyond the test set; and only more train­ing runs sup­port a claim about a recipe: in deep RL, a 25% improve­ment in median score needed about 25 runs per algo­rithm to be sta­tis­ti­cally defen­si­ble [163]. Low­er­ing the sam­pling tem­per­a­ture to cut vari­ance changes what you mea­sure [5], and the serv­ing stack is a draw of its own: one model's AIME 2024 score varied by up to 8 points across com­pute clus­ters [164]. Run both arms on one stack, inter­leaved.

Pair, cluster, then count

Can the same items buy more pre­ci­sion? Yes, for free, if both arms answer them. The per-item dif­fer­ence has stan­dard error √(SE_A² + SE_B² − 2ρ·SE_A·SE_B), where ρ, the cor­re­la­tion of the two arms' per-item scores, is usu­ally pos­i­tive because models tend to agree on which items are easy and which are hard [5]; at ρ = 0.5, pair­ing halves the vari­ance of the dif­fer­ence (com­puted). Check­ing whether two 95% inter­vals over­lap is the wrong test: with no true dif­fer­ence and equal errors they over­lap more than 99% of the time [166], so the check misses most real gains.

Clus­ter on what you gen­er­ated. Vari­ants of one seed share its quirks, and episodes in one envi­ron­ment share its bugs; clus­tered stan­dard errors were 3.05 times the naive ones on DROP [5]. Kish's design effect says vari­ance grows by 1 + (m − 1)ρ for clus­ters of m items with intra-clus­ter cor­re­la­tion ρ [167], so ten vari­ants per seed at ρ = 0.3 make 1,000 tasks worth about 270 inde­pen­dent ones, and because the effec­tive count k·m/(1 + (m − 1)ρ) approaches k/ρ, no number of vari­ants lifts 100 seeds past about 333 (com­puted). Boot­strap the whole hier­ar­chy too: in a sim­u­la­tion cal­i­brated to a com­puter-use bench­mark, resam­pling only roll­outs gave a nom­i­nal 95% inter­val "just 17% suite-level cov­er­age" [168].

Then count before you run. The items needed to detect a gap δ are n = (z_α/2 + z_β)²·V/δ², where V is the per-item vari­ance of the dif­fer­ence; under Miller's illus­tra­tive assump­tions a 3-point gap needs about 969 ques­tions, hence his advice of at least 1,000 for a new eval­u­a­tion [5]. Rare events are worse: telling a reward-hack rate of 0.093% of episodes from 0.046%, a pair one lab reports for two recent models [169], takes about 49,000 inde­pen­dent episodes per model at 80% power (com­puted). Figure 25 does this arith­metic for your design.

Games build pair­ing into the format. Dupli­cate Scrab­ble deals every player the same tiles [170], Stock­fish's test farm scores games in pairs [171], and poker's AIVAT sub­tracts the luck of the cards with a value func­tion, which "can reduce the number of hands needed to draw sta­tis­ti­cal con­clu­sions by more than a factor of 10" [172]. Pair­ing works only while the shared luck stays shared: in the tile-game test­bed, play­ing each shuf­fled bag twice with seats swapped cut the stan­dard devi­a­tion of a pair's result by only 7% to 13% against two inde­pen­dent games, because the draws diverge after the first dif­fer­ent move. A switch-on design works far better. When a com­po­nent acts only late, as an endgame solver does, both arms play iden­ti­cal moves until it switches on, and the per-pair stan­dard devi­a­tion fell from about 0.31 to 0.07⁠–⁠0.09, cut­ting the pairs needed 11- to 22-fold (test­bed mea­sure­ment). When a com­par­i­son comes back even, state the null as an inter­val: a 1,000-game match that ends at 50% rules out only gains beyond about ±22 Elo (com­puted).

Look past the mean

What does a mean hide? Two tails: the answers a model should not have given, and the tasks it cannot do reli­ably. Train­ing that climbs the tail, toward reli­a­bil­ity on every attempt and on the hard­est tasks, is invis­i­ble to a mean, and so is train­ing that qui­etly makes a model guess more.

Start with wrong versus abstained. Under binary grad­ing "I don't know" scores the same as a wrong answer, so "IDK-type responses are max­i­mally penal­ized while an over­con­fi­dent 'best guess' is opti­mal" [79]. On SimpleQA, GPT-4o is cor­rect on 38.2% of ques­tions and wrong on 60.8%; Claude-3.5-Sonnet is cor­rect on 28.9%, abstains on 35.0% and is wrong on 36.1% [173]. Ranked by accu­racy GPT-4o beats Claude-3.5-Sonnet; ranked by wrong-answer rate the order reverses. Kalai et al. pro­pose stat­ing a con­fi­dence target t in the prompt: +1 for a cor­rect answer, 0 for abstain­ing and −t/(1 − t) for a mis­take. Under it, GPT-4o stays ahead of Claude only while t is below about 0.27; o1-pre­view (42.7% cor­rect, 48.1% wrong) beats GPT-4o at every t; and from t ≈ 0.47 a model that always abstains beats all three (com­puted; Figure 26). The penalty belongs in the report, and the same rule is the reward that teaches a model to abstain.

Then the reli­a­bil­ity tail. τ-bench's pass^k is the chance that all k attempts at a task suc­ceed; its GPT-4o agent suc­ceeded on about 61% of retail tasks per attempt and below 25% at pass^8 [14]. A mean cannot tell you which world you are in: two agents with a 60% mean have pass^8 of 1.7% if every task suc­ceeds 60% of the time, or 60% if 60% of tasks always suc­ceed and the rest never do (com­puted). METR's strong­est mea­sured agent has a 50% time hori­zon of about 17 hours (an esti­mate METR flags as unre­li­able) but an 80% hori­zon of about 3.1 hours [174], and on Harvey's Legal Agent Bench­mark a fron­tier model passed 91.2% of rubric cri­te­ria but com­pleted only 8.3% of tasks [169]. Cri­te­ria passed and tasks fin­ished are dif­fer­ent prod­ucts.

The bottom of the task dis­tri­bu­tion has its own selec­tion trap. If every task truly suc­ceeds 60% of the time and you rank tasks by four attempts each, the worst tenth aver­ages about 19% (com­puted), because pick­ing the worst tasks also picks their unlucky draws. A cur­ricu­lum that retires tasks on a short streak, or a report of "our gain on the hard­est decile", inher­its this regres­sion to the mean; choose the hard band on one set of attempts and score it on fresh ones.

Fair comparisons

How does an eval­u­a­tion flat­ter one side when nobody cheats? Three ways: the items were screened against a rival, the rival did not get its best effort, and one side was selected from many tries while the other ran once.

The screen. Adver­sar­ial fil­ter­ing keeps the items a ref­er­ence model gets wrong, so that model and its rel­a­tives score low on what­ever remains; Phang et al. found rank­ings on fil­tered sets "highly sen­si­tive to the choice of adver­sary model", and "There is no case where the rank improves when fil­ter­ing with the same pre­trained model" [175]. SimpleQA, for exam­ple, "is adver­sar­i­ally col­lected against GPT-4 responses" [173]. In the toy of Figure 27, models A and B are equally able, yet A leads by about 10 points on the items B got wrong at screen­ing and by noth­ing on an unscreened sample. The screen also sets a regres­sion trap: B scored 0% on its kept items by con­struc­tion but about 39% when run again, so a "new ver­sion" of B shows a large, spu­ri­ous gain on its own fil­tered set (toy sim­u­la­tion).

Beat the oppo­nent at its best. Well-tuned LSTMs beat newer archi­tec­tures once both sides were tuned [176]. For RL on lan­guage models, best effort includes test-time com­pute: "base models achieve higher pass@k score when k is large" [157], so com­pare at matched sam­pling com­pute with a selec­tor you could deploy, such as a major­ity vote or a ver­i­fier, never an answer key. Chess shows the stan­dard: after crit­ics objected to the 2017 match con­di­tions, the rematch gave Stock­fish three hours a game, a 32 GB hash and endgame table­bases, and AlphaZero still scored +155 −6 =839 over 1,000 games [177].

Selec­tion is the quiet asym­me­try. Before the Llama 4 release, Meta tested 27 pri­vate vari­ants on Chat­bot Arena, and such best-of-N dis­clo­sure "sys­tem­at­i­cally inflates model rank­ings" [165]. Pick­ing the best step of one train­ing run is the same move: the best of 10 equally good check­points reads 7.5 points high on the 100-item set it was chosen on and 1.7 points high on 2,000 items, and a fresh con­fir­ma­tion set removes the gain (com­puted exactly).

Luck ceilings in stochastic, hidden-information domains

What is the best any legal player could do? With chance and hidden state the answer is not 100%, and a "gap to per­fect" that no legal player can close looks like a mod­el­ing fail­ure and absorbs work that buys noth­ing. The tile-game test­bed mea­sures the ceil­ing directly; W is the win rate with ties counted as half.

Luck is large, and it is made during play. In 20,000 games of a greedy bot (always take the high­est score now) against itself, where every point of spread is luck, the final spread had a stan­dard devi­a­tion of 96 points, and the player who drew both blank tiles, the game's two wild­cards, won 73% of those games (test­bed mea­sure­ment). A player that outscores its oppo­nent by Δ points a game on aver­age wins about Φ(Δ/σ), with Φ the stan­dard normal dis­tri­bu­tion func­tion, so at σ ≈ 96, win­ning 90% of games would need a mean lead of about 123 points (com­puted); the best legal player mea­sured led the greedy bot by 65 and won 79%. Public work sees the same scale: a player that could see its oppo­nent's rack outscored its rival by 37 points a game [16].

The ceil­ing ladder in Figure 28 gives each rung a mea­sured win rate, here against the greedy bot. The best legal player won 79.3% [78.1, 80.5], and the same search design at two to three times the com­pute did no better. Shown the tiles the oppo­nent kept from its last move, the most any rack infer­ence could ever recover, it gained noth­ing (78.7%); shown the oppo­nent's whole rack it won 92.8%, and shown the bag order too, all 400 of its games. The ille­gal rungs use the same search design, so each is a lower bound on what that infor­ma­tion is worth and the close­ness fig­ures that follow are upper esti­mates. Close­ness, (W − 50%)/(bound − 50%), depends on the oppo­nent: 0.68 against the greedy bot, 0.40 against the equity bot (63.7% against 84.3%) and 0.23 against the sim­u­la­tion bot (54.2% against 68.5%). The weaker the oppo­nent, the closer legal play sits to its infor­ma­tion bound; against the strong­est one, three quar­ters of the reach­able dis­tance is still open. Against the greedy and equity bots (the kept-tiles rung was not run against the sim­u­la­tion bot), the 95% lower bounds put at least about 80% of the gap to the rack-seeing bound on tiles the oppo­nent has just drawn, which noth­ing it does can reveal (test­bed mea­sure­ment). A public exper­i­ment found the same shape: Bayesian rack infer­ence won 51.9% ± 0.8% against a strong bot, know­ing the oppo­nent's exact kept tiles 53.3%, and a full-rack oracle only about 62% [6].

Inside the bound, search and learn­ing paid in an unex­pected order (all test­bed mea­sure­ments). More of the same search stopped paying, at about +2 points of W per dou­bling of sim­u­la­tions, and four times the sim­u­la­tions at a short hori­zon added noth­ing, while one more ply of look-ahead added 4.0 points [1.4, 6.6]: more sam­ples of a biased esti­mate only make the bias more cer­tain. The reli­able gains came from exact­ness and a better value. An exact solver for the per­fect-infor­ma­tion endgame added 1.9 points of W, and a win-prob­a­bil­ity model that knew whose turn it was cut held-out log-loss from 0.4835 to 0.4516 and added 2.25 points [0.3, 4.2].

Self-play has a limit of its own. Expert iter­a­tion trains a net­work on a search player's choices and lets it guide the next search [144], the loop behind AlphaZero [178]. In the test­bed it played about half a mil­lion games, improved once by roughly 5 Elo and plateaued: the appren­tice caught up with the expert in one gen­er­a­tion, and a sharper copy of the same search is not a stronger player. Self-play com­pounds only when each gen­er­a­tion's expert can find some­thing the last could not, which a learned value sup­plies and KataGo's play­out-cap ran­dom­iza­tion was built to feed [179]. Sequen­tial pro­mo­tion tests are weak evi­dence on their own: in the test­bed, one passed a can­di­date truly worth +2 Elo 11% of the time, so con­firm every pro­mo­tion on a fresh, fixed-size match (test­bed mea­sure­ment).

Markets: the environment must carry the statistics

Why are mar­kets the hard­est RL domain? Because every prop­erty that makes an envi­ron­ment trust­wor­thy fails there at once.

1. Most of every reward is luck. The best machine-learn­ing fore­casts of indi­vid­ual US stock returns reach a monthly out-of-sample R² of 0.33% to 0.40% [180]. An annu­al­ized Sharpe ratio esti­mated from T years has a stan­dard error of roughly 1/√T [181], so a true Sharpe of 0.7 needs about 8 years to reach t = 2, and one quar­ter of daily data has a stan­dard error near 2 (com­puted).
2. The reward drifts. Across 97 pub­lished pre­dic­tors, port­fo­lio returns were 26% lower out of sample and 58% lower after pub­li­ca­tion [182].
3. The agent moves the market. Impact grows roughly with the square root of the size traded [183], so ignor­ing the agent's own orders rewards strate­gies that work only at zero size.
4. Sim­u­la­tors are wrong where it mat­ters. Replay "does not sub­stan­tially adapt to or respond to the pres­ence of the exper­i­men­tal strat­egy", and for inter­ac­tive agent-based sim­u­la­tion "it is unclear if the market envi­ron­ment they pro­vide is real­is­tic" [184].
5. Lan­guage models remem­ber the future. Told to ignore every­thing after 2010, GPT-4o still called quar­terly GDP growth against a 2.5% thresh­old with 98.0% accu­racy on the quar­ters after it, against 97.6% before [185].

So the envi­ron­ment, not the agent, must carry the sta­tis­tics. Five design rules follow.

Make an as-of API the only data path, gated on knowl­edge time. Every record car­ries three clocks: valid time (the period a fact describes), knowl­edge time (when a par­tic­i­pant could first have known it) and cap­ture time (when your pipe­line saw it) [186] [187], and only knowl­edge time answers "could the agent have known this at the as-of time?" The SEC's frames API "aggre­gates one fact for each report­ing entity that is last filed" [188], so it serves the value from the latest filing that reports the period: 4,195 of the 4,759 diluted-EPS facts for cal­en­dar Q2 2025 (88%) come from 2026 fil­ings, mostly the next year's com­par­a­tives, whose values can differ from what was know­able in August 2025 (com­puted from the public API on Octo­ber 18, 2026). Today's state can also encode tomor­row's label: in vul­ner­a­bil­ity triage, a naive random split with unfil­tered evi­dence inflated appar­ent prospec­tive recall 8.5-fold [189]. Split in time order, purged and embar­goed when labels span time [190].

Score every can­di­date on shared random draws. Any (PO)MDP can be made deter­min­is­tic by draw­ing its random num­bers up front and reusing them for every policy [191]. In the toy exe­cu­tion sim­u­la­tor, detect­ing a 0.23 basis-point improve­ment at 80% power took about 3.15 mil­lion trad­ing days on unpaired real­ized costs, about 5,400 paired on the same days and shocks, and about 37 once the envi­ron­ment replaced real­ized price luck with its known expec­ta­tion (toy sim­u­la­tion; Figure 29). Live, where two poli­cies cannot trade the same order, ran­dom­ized orders need about 7,400 orders, and the usual mid-price bench­mark for­gives the trader's own lin­ger­ing impact, which in the toy reversed two true rank­ings of sched­ules. Prefer expected scores where out­comes resolve late: in public work, reward­ing "the real­ized per-play out­come fails" [192].

Keep a trial ledger and charge the reward for the search. The expected best in-sample Sharpe of 100 skill-less six-year back­tests is about 1.03 (com­puted with Bailey and López de Prado's for­mula [193]), so "Sharpe 1.0, best of 100" is what pure search pro­duces. In a toy research gym (Figure 29), the naive in-sample reward paid 1,000 noise ideas more than ten honest ones, while White's Real­ity Check [194], boot­strapped over the whole ledger, passed noise at its nom­i­nal 5%.

Put costs and impact inside the tran­si­tion, and anchor money rewards to a cer­ti­fied bound on real data. Score an exe­cu­tion or stor­age-arbi­trage policy as a cap­ture ratio against a per­fect-fore­sight opti­mum on the same prices: the reward never sat­u­rates, any score above the bound is a bug, and the bound sizes the prize (in the toy exe­cu­tion sim­u­la­tor, 0.24 basis points over the closed-form sched­ule).

Check live base­lines and licenses before the archi­tec­ture. Most models in StockBench strug­gled to beat buy-and-hold [195], two of seven models made money in PolyBench [196], and in LiveTradeBench high LMArena scores did not imply better trad­ing [197]. Licenses bind before prices: Kalshi's data terms forbid train­ing an AI system on its data with­out con­sent [198], and from the US con­sol­i­dated tape's planned oper­a­tive date of April 1, 2027, train­ing a model on tape data is non-dis­play use [199].

The check­list that fol­lows col­lects every rule by the ledger line it pro­tects.

The practitioner checklist

What is the whole guide as a list you can run on Monday?

The deci­sion rules of Parts I to VI, grouped by the ledger line each pro­tects, each with one anchor­ing number. Use it to spec­ify an envi­ron­ment, audit a run, or read some­one else's claimed gain.

0. Instru­ment (Part I)

1. Iso­late every­thing that scores from every­thing being scored, infra­struc­ture included, and re-test with each model gen­er­a­tion (an exploit agent that did no legit­i­mate work scored 73⁠–⁠100% on eight bench­marks).
2. Prove sound­ness before learn­abil­ity: oracle at the max­i­mum, do-noth­ing far below, every canary at or below the floor (do-noth­ing beat GPT-4o on τ-bench air­line, 38% to 35%).
3. Design refresh into the first release, and reward a ratio where one exists (tar­geted bench­marks passed 80% within 9⁠–⁠16 months).

1. Empty (Part II)

1. Keep tasks in the band of the policy you train, and refill it as it drains (at p = 0.9 and 8 roll­outs, 43% of groups carry no gra­di­ent).
2. Test the ver­i­fier before gen­er­at­ing: ref­er­ence passes, known-wrong fails, refor­mat­ted-cor­rect passes, degen­er­ate fails (rule check­ers miss 14% of cor­rect answers).
3. Give every oper­a­tor a stated con­tract, and mea­sure its effect on pass rate and on one short­cut fea­ture.
4. Esca­late roll­outs sequen­tially before retir­ing a task (ruling out a 2% pass rate takes 149 straight fail­ures).
5. Dedu­pli­cate and split on a struc­tural sig­na­ture, and drop whole lin­eages of seeds that match eval­u­a­tion items (rephrased test items lifted one model from 28.7 to 95.3).
6. Report real-task scores beside syn­thetic ones, buy seed fam­i­lies before vari­ants, and price a usable task through the funnel, not the LLM bill.

2. Wrong (Part III)

1. Judge a false pos­i­tive by whether the policy can pro­duce it on pur­pose across prompts (test-skip­ping hacks spread to nearly every envi­ron­ment of one fron­tier run).
2. Bound the false-pos­i­tive rate on a fixed adver­sar­ial set, denom­i­na­tor stated (zero in 299 gold-wrong sam­ples bounds it below 1%).
3. Treat dif­fi­culty cal­i­bra­tion as an integrity con­trol (a lone hacked sample in a sat­u­rated group of 8 gets advan­tage √7).
4. Never clip a par­tial-credit reward at the floor, and test that every canary trips its own defense (clipped, the Brier score pays 0.70 for false cer­tainty against 0.448 for the truth).
5. Judge an exploit policy by its mean, gate detected hacks or price the penalty above bonus ÷ recall, and offer an abort action (cheat­ing on impos­si­ble tasks fell from 54% to 9%).
6. Put judges behind a strict rule, ask narrow ques­tions in both orders, and audit the judge's false-pos­i­tive rate on rejects (a ref­er­ence answer cut judge fail­ures from 14 of 20 to 3 of 20).
7. Hold out a gold signal for any learned reward and stop by diver­gence against √KL.
8. Score the same out­puts a second way and run a random-reward con­trol on a second model family (re-scor­ing moved models 200 or more leader­board places).

3. Cor­rupted (Part IV)

1. Mea­sure trainer-idle, gen­er­a­tor-idle and p99 roll­out time before redesign­ing (roll­out took 63⁠–⁠87% of pro­duc­tion steps).
2. Cor­rect stal­e­ness and mis­match with the sam­pler's recorded log-prob­a­bil­i­ties, and alert on the sam­pler-trainer log-ratio (naive PPO 31.6, decou­pled 40.6 at stal­e­ness 4 to 16).
3. Check trained against gen­er­ated length dis­tri­bu­tions under fast sched­ulers; size warm pools by L + z√L; key steps by request id; mask infra­struc­ture fail­ures.
4. Train on the sam­pler's token ids with a writer mask, a con­stant loss divi­sor, masked trun­ca­tions and con­text resets as seg­ments (Search-R1: 0.431 masked against 0.343 unmasked).

4. Coarse (Part V)

1. Record the recipe, not the trainer's name, and add a piece only if the mech­a­nism it acts on is present.
2. If an oracle can score every option cheaply, dis­till its rank­ing with a soft target (given larger teach­ers, 74.4 for 1,800 GPU-hours against 67.6 for 17,920).
3. Keep RL for ver­i­fier-only domains, priv­i­leged teach­ers, myopic ora­cles, beat­ing the teacher and expen­sive scores, and buy on-policy labels only when the stu­dent drifts.
4. Put knowl­edge in through tokens and tools, use RL for reli­a­bil­ity, and hand exactly solv­able sub-prob­lems to solvers.

5. Unread­able (Part VI)

1. Size the eval­u­a­tion before you run it (a 3-point gap needs about 1,000 paired ques­tions under Miller's illus­tra­tive assump­tions).
2. Report Wilson or boot­strap inter­vals on small sets; pair on shared items and clus­ter on the seed (100 seeds cap near 333 effec­tive tasks at ρ = 0.3).
3. State a null as an inter­val (an even 1,000-game match rules out only gains beyond ±22 Elo).
4. Report cor­rect, wrong and abstained with the penalty, pass^k, and a worst decile scored on fresh attempts (a 60% mean allows pass^8 from 1.7% to 60%).
5. Name every screen, give the oppo­nent its best effort, and report the vari­ants you tried (best of 10 equal check­points reads 7.5 points high on 100 items).
6. In mar­kets, gate data on knowl­edge time, pair on shared draws, charge for the trial ledger and score against a cer­ti­fied bound (the best of 100 noise back­tests shows a Sharpe near 1.03).

6. Absent (Part VI)

1. Mea­sure an ille­gal-infor­ma­tion bound against the oppo­nent you care about before set­ting a goal, and judge close­ness as a share of it (a full-rack oracle won only about 62% against a strong bot).
2. Spend noth­ing on infor­ma­tion that does not exist at deci­sion time (in the same public test, rack infer­ence won only 51.9%).

Most of these checks have open imple­men­ta­tions to start from (Table 12); read each one's defaults against the rules above.

Then attach an eval­u­a­tion card to every RL result (Table 13). Each row answers a dif­fer­ent ques­tion about whether the loop learned.

Finally, the signal ledger, filled in (Table 14).

An envi­ron­ment cannot make a domain less noisy or a hidden tile vis­i­ble. What it can do is make every number it emits honest about how much search, luck, leak­age and noise stand behind it: the trials it ran, the draws it shared, the clock it gated, the bound it mea­sured. An envi­ron­ment that reports those four things is a mea­sur­ing instru­ment. One that does not is a source of num­bers an opti­mizer will learn to please.

Methods: the toys and testbeds

What stands behind the toy and test­bed num­bers?

Each toy and test­bed is an unpub­lished pro­gram the author wrote and ran, so its num­bers cannot be checked against an out­side source (Table 15). Inter­vals over three seeds are Stu­dent t-inter­vals with two degrees of free­dom, about 2.2 times wider than a normal inter­val with the same stan­dard error.

Notes

1. Scrabble is a trademark of Hasbro, Inc. in the United States and Canada and of Mattel, Inc. elsewhere. The testbed is an independent implementation of a two-player crossword tile game with a shuffled bag and hidden racks; it is not affiliated with either company. ↩
2. With the population standard deviation, a group of eight with one bonus b has mean b/8 and standard deviation b√7/8, so the hacked sample's advantage is (7b/8) / (b√7/8) = √7. With the sample standard deviation it is about 2.47; the cancellation of b is the same. ↩
3. The number of slots refilling at any moment is roughly Poisson with mean L, so L + z√L covers its upper quantile: the square-root staffing rule of queueing theory. Erlang B, the exact loss formula, gives 77 here. ↩