RL environments as a business

In 2026 two fron­tier labs pub­lished audits of their own mea­sur­ing sticks. Anthropic froze all changes to its pro­duc­tion RL envi­ron­ments for roughly a month, “flagged over 10% of envi­ron­ments in our pro­duc­tion mix”, and required “every fixed envi­ron­ment to be re-cer­ti­fied before it could enter another train­ing run”[1]. OpenAI audited 138 SWE-bench Ver­i­fied prob­lems that o3 did not con­sis­tently solve over 64 runs, found that 59.4% of them (a rate on a fail­ure-selected subset, not the whole bench­mark) “con­tained mate­rial issues in test design and/or prob­lem descrip­tion”, and stopped report­ing the bench­mark[2]; on its suc­ces­sor it esti­mates that “~30% of SWE-bench Pro tasks are broken”[3]. These are the best-resourced buyers of RL envi­ron­ments and evals in the world, and the num­bers they relied on were nois­ier and more broken than anyone out­side could see.

This arti­cle is about the busi­ness that fol­lows. Its argu­ment is that the money in RL envi­ron­ments goes to who­ever can make a score mean some­thing. A score can be wrong (scor­ers, wrong answer keys, leaks and selec­tion), unre­peat­able (run-to-run noise nobody mea­sured) or blind to the rare, costly cases (test sets mirror traf­fic rather than loss), and rein­force­ment learn­ing turns each fail­ure into trained behav­iour. Eight busi­ness areas attack these fail­ures; pressed against their strong­est objec­tions, almost every one ends up sell­ing a ver­dict rather than a model. That con­ver­gence is the oppor­tu­nity.

It is writ­ten for founders, oper­a­tors, investors and buyers, and assumes only that you know what fine-tuning and a bench­mark are. Every exter­nal fact links to a public source, tagged where it is a party’s own account (self-reported), sec­ond­hand (reported) or this arti­cle’s arith­metic (esti­mate, derived); toy sim­u­la­tions are labelled and are never data, and so is one unpub­lished exper­i­ment by the author. Facts are as of 18 Octo­ber 2026; noth­ing here is invest­ment advice. A com­pan­ion arti­cle, RL envi­ron­ments: a field guide, covers how to build the envi­ron­ments them­selves.

Part 1The market: who pays, for what, and how

Where is the money, who signs, what is the unit of value, and what does one unit cost to make? In short: fron­tier labs pay, a researcher decides before pro­cure­ment does, the unit is a ver­i­fied task rather than an envi­ron­ment, and a bespoke task often costs more to make than labs pay for it.

What a lab is actually buying

An RL envi­ron­ment has three parts: a task (or a world that hosts many tasks), the tools and state an agent acts on, and a ver­i­fier that runs out­side the agent’s reach and returns a reward. If the agent can read, edit or imi­tate the check, the reward mea­sures its access to the checker, not its skill.

Labs pay for the ver­i­fied task: a prompt, a sound grader, a ref­er­ence solu­tion and a mea­sured pass rate against cur­rent models. “Labs are not buying envi­ron­ments, they are buying tasks,” as Chakra Labs puts it[4]. A world that hosts hun­dreds of tasks and sur­vives sev­eral model gen­er­a­tions prices like a soft­ware licence; a coding task that sat­u­rates within a year prices like con­tent.

Demand scales with con­cur­rency: Qwen3-Coder ran 20,000 envi­ron­ments in par­al­lel[5]. But DeepSeek-V3.2’s own pipe­line syn­the­sised 1,827 task envi­ron­ments[6], the first warn­ing for a vendor: when a lab’s agents can write envi­ron­ments that are cheap to verify, nobody pays for cheap ver­i­fi­ca­tion. What labs lack is qual­ity at volume: “Main­tain­ing qual­ity while scal­ing is the number one bot­tle­neck that people see”[7].

How much money, measured how

The head­line fig­ures dis­agree because they mea­sure dif­fer­ent things, and read­ing one as another can inflate a plan ten­fold (Figure 1). Budget talk is large: Anthropic’s lead­ers “dis­cussed spend­ing more than $1 bil­lion on RL envi­ron­ments over the next year” (reported)[8]. Esti­mated actual spend is far smaller: Wing VC put Anthropic’s spend­ing with coding and com­puter-use envi­ron­ment ven­dors “on the order of tens of mil­lions annu­ally”[9], and another report put envi­ron­ments below 10% of Anthropic’s post-train­ing budget (reported)[10]. Vendor rev­enue is gross: 28 train­ing-data star­tups bill about $8.5B a year, over three quar­ters of it at Scale, Surge, Mercor and Hand­shake (reported)[11], mostly expert pay passed through: Mercor’s $2B+ run rate is about $600⁠–⁠800M “net of the experts who do the work”[12]. Com­pute dwarfs every­thing: OpenAI’s 2026 R&D com­pute is pro­jected at about $19B, and “com­pute spend­ing will still dwarf RL envi­ron­ment spend­ing”[7].

This arti­cle’s order-of-mag­ni­tude esti­mate is $0.5⁠–⁠3B a year of lab spend­ing on envi­ron­ments and tasks by mid-2026, built bottom-up and drawn as its own row in Figure 1 (esti­mate).¹ Two con­se­quences follow. A market slide that mul­ti­plies $8.5B by a share is mostly mul­ti­ply­ing pass-through labour. And because envi­ron­ment spend is a small frac­tion of the com­pute it steers, the strong­est price argu­ment is the buyer’s com­pute bill, not the vendor’s cost. Train­ing is the bigger pool: labs use bought envi­ron­ments for RL “per­haps 10⁠–⁠20x” more than for bench­mark­ing, one startup employee told Epoch[7].

Who signs, and in what order

No fron­tier lab pub­lishes a request for pro­pos­als for envi­ron­ments, and buying runs in the oppo­site order from enter­prise soft­ware: sci­ence first, paper second. Anthropic’s RL data team aims to take “an idea for new data on Monday” to “the train­ing mix by Friday”[14]; pro­cure­ment arrives once a researcher already wants the work, then nego­ti­ates “rate cards, accep­tance cri­te­ria, change con­trols, audit rights, and ter­mi­na­tion pro­tec­tions”[15] and keeps “qual­i­fied second sources” in every crit­i­cal cat­e­gory[16].

Three facts in the table decide most deals. The ver­i­fier is tested before price is dis­cussed: Epoch’s inter­vie­wees “con­sis­tently cited robust­ness against reward hack­ing as a key qual­ity cri­te­rion”[7]. So a sample ships its own proof: the ref­er­ence solu­tion reaches full reward, a do-noth­ing agent and known hacks score at the floor, and pass rates sit above a floor of “around 2⁠–⁠3%”, with at least one suc­cess in 64 or 128 roll­outs, and below the roughly 70% at which labs dis­card a task. Secu­rity can end an account overnight: after a breach through a 40-minute com­pro­mise of an open-source depen­dency, “Meta has paused its con­tracts with Mercor indef­i­nitely” (reported)[19]. And because multi-sourc­ing is the default, a new vendor’s real­is­tic first goal is to become a qual­i­fied second source.

What it sells for

No vendor pub­lishes a price sheet, but inter­views and fil­ings give a usable ladder (Figure 2). Most tasks sell for “$200⁠–⁠$2,000”; “$20k per task would be rare but pos­si­ble”; a high-fidelity clone of a prod­uct like Slack runs around $300k; con­tracts are “often six to seven fig­ures per quar­ter”, around $300k–$500k at one neolab; and “exclu­sive deals are roughly 4⁠–⁠5x more expen­sive than non-exclu­sive ones”[7]. Labs “are will­ing to pay $5000+ for a single decent coding task”[20], and web­site repli­cas “often cost about $20,000 per web­site”, with one lab buying hun­dreds[21].

The most useful number on the ladder is not a price. Mech­a­nize, which sells envi­ron­ments, esti­mates that a task steers about $480 of com­pute per train­ing run and $2,400 over its life at retail API prices, and calls it “inef­fi­cient for AI labs to spend sig­nif­i­cantly less than ~$500 per RL task”[22].² Treat it as a ceil­ing, since a lab pays less than retail per token; the direc­tion still holds. A broken task at $200 is expen­sive, because the lab trains on a signal that teaches the wrong thing and pays in com­pute and in trained-in behav­iour. And price is capped by how ver­i­fi­able and scarce a task is long before it reaches what the work is worth: real free­lance soft­ware tasks paid $50 to $32,000[23].

What one accepted task costs to make

Com­pute is the small line; labour and yield, the share of drafts that sur­vive cal­i­bra­tion and review, are the big ones. This arti­cle’s cost model (an esti­mate from stated assump­tions inside public ranges) puts an accepted bespoke task at $1,606⁠–⁠$2,501 in law, soft­ware, medicine and finance, about $6,485 in secu­rity and about $21,379 at fron­tier-grade soft­ware; tem­plate-gen­er­ated tasks cost $327⁠–⁠$396 and tasks mined from real arti­facts about $161. Com­pute is 3⁠–⁠11% of a bespoke task’s cost, against 27% for tem­plates and 61% for mined tasks.³ A public cross-check agrees: Mercor spent more than $500,000 on a 200-task bench­mark, about $2,500 a task[24]. The quoted $200⁠–⁠$2,000 band is there­fore a tem­plate-and-mining market with a bespoke tail: a vendor that only writes bespoke tasks must price above the band, sell exclu­siv­ity, or lose money.

Deal shape, not price alone, decides the margin

Run the same assets through dif­fer­ent deal shapes and mar­gins spread from a loss to soft­ware levels (Figure 3, same cost model):

• Exclu­sive bespoke at the top of the quoted band loses money. A finance envi­ron­ment licensed for $150,000 plus 300 expert tasks at $2,000, sold exclu­sively to one lab, runs at −26%; at $5,000 a task it earns 43%. Break-even is about $2,660 a task, above the whole public band.⁴
• Resale car­ries the margin. Sell the $5,000 deal’s assets non-exclu­sively to two more labs after a six-month window, at 25% of the first order, and the new rev­enue earns 95% (about 60% com­bined). That requires keep­ing the rights, as Prox­i­mal’s public ser­vices agree­ment does, licens­ing cus­tomers on “a lim­ited, non-exclu­sive, non-trans­fer­able” basis[25].
• A cat­a­logue needs two buyers before it exists. A 1,000-task tem­plate library sold to three labs at $500 a task earns 62%; with one buyer it loses 5%.
• Refresh at bespoke cost is a trap. A year-two refresh at $400,000 a lab from three labs earns 6% with bespoke replace­ments and 58% with tem­plate ones, and model improve­ments wash out about a third of tasks a year[26].

Yield is the lever a vendor con­trols. Rais­ing cal­i­bra­tion yield from 0.40 to 0.80 cuts the accepted finance task from $3,690 to $2,144. Paying experts $250 an hour instead of $100 moves it from $1,817 to $3,867, but the labour market sets that rate. Checks that kill bad drafts before cal­i­bra­tion pay back first.

Public gross mar­gins sort the busi­ness models the same way. Expert mar­ket­places run thin, Mercor at 27% in 2025 and 33% in the second quar­ter of 2026 (reported)[27]; man­aged data with a labour markup earns “50%+” (Scale, esti­mate)[28]; resold off-the-shelf datasets reach “as high as 80% to 90%” (micro1, reported)[29]; young RL-as-a-ser­vice firms “often lose money”[21]. Val­u­a­tions echo it only weakly: Prox­i­mal’s reported $15 mil­lion seed at a $300 mil­lion val­u­a­tion beside $200 mil­lion of annu­alised rev­enue is a low ratio if that run rate is gross, which the report does not say[30]. Reuse, not exclu­siv­ity or head­count, moves a vendor out of the ser­vices bracket.

Post-training is becoming a rentable commodity

The train­ing step is losing its price. OpenAI is wind­ing down self-serve fine-tuning: from 6 Jan­u­ary 2027 “Active exist­ing cus­tomers will no longer be able to create new fine-tuning jobs”, and fine-tunes “follow the same shut­down sched­ule as their under­ly­ing base models”[31]. At Fire­works, “Man­aged Train­ing no longer accepts new RFT jobs”[32]. What spread instead is a shape: Tinker’s four train­ing prim­i­tives[33] reap­pear in Fire­works’ “Tinker-com­pat­i­ble train­ing client”[34] and Micro­soft Foundry’s inter­ac­tive train­ing pre­view[35]. Each sells the loop and expects the cus­tomer to bring the envi­ron­ment and the grader, while base models churn under­neath: Tinker retired 19 models on 12 June 2026 and advises “Avoid build­ing hard depen­den­cies on any spe­cific model”[36]. The durable assets are the envi­ron­ment, the grader, the cleaned labels and the recipe; a vendor that sells only train­ing sells a com­mod­ity with a dep­re­ca­tion clock.

What acquirers pay for

Acquir­ers pay for a posi­tion on traf­fic, teleme­try or data more than for a stand­alone tool. Rubrik bought Pred­ibase, a 25-person fine-tuning com­pany, for over $100 mil­lion[37], while exper­i­men­ta­tion and observ­abil­ity posi­tions fetched far more: OpenAI paid $1.1 bil­lion in stock for Stat­sig[38] and Dyna­trace agreed to pay $915 mil­lion for Arize[39]. Explain­ing Data­dog’s pur­chase of his com­pany, Adap­tive ML’s CEO pointed to “the con­tin­u­ous stream of real-world sig­nals” only such a plat­form pro­vides: “The miss­ing piece was never the algo­rithm”[40]. One deal deserves a second look: OpenAI acquired the eval tool Prompt­foo[41], and its dep­re­ca­tions page sends devel­op­ers leav­ing its own Evals prod­uct, read-only from 31 Octo­ber and shut on 30 Novem­ber 2026, to “Moving from OpenAI Evals to Prompt­foo”[31]. The party whose models are mea­sured now steers its cus­tomers’ evals onto a tool it owns, which leaves room for a mea­surer with no stake in the answer.

Concentration and neutrality

A lab cus­tomer can leave faster than a vendor can replace it: in Jan­u­ary 2024 Google gave Appen about two months’ notice that it would end a con­tract worth roughly a third of Appen’s rev­enue[42], and Appen’s shares fell more than 40% in a day[43]. Neu­tral­ity is part of the prod­uct: after Meta bought 49% of Scale AI, Google, which had planned to pay Scale about $200 mil­lion that year, was reported to be plan­ning to cut ties[44].

Where supply crowds and where it thins

Crowded: coding, com­puter use and generic enter­prise work­flows. Of the 42 ven­dors in the RL List direc­tory, 23 carry its Coding tag and 19 Com­puter Use[45], and open hubs such as the Envi­ron­ments Hub Prime Intel­lect opened in August 2025[46] price generic envi­ron­ments toward zero. Thin: chip design, ERP back office, fund oper­a­tions, anti-money-laun­der­ing, secu­rity-alert triage and insur­ance claims; none of the 42 records men­tions fund oper­a­tions, anti-money-laun­der­ing or secu­rity-alert triage (derived). Thin supply is an open­ing only where a ver­i­fier can be built that nobody can argue with. The same holds for raw data: pri­vate archives now sell openly (Google won a bank­ruptcy auc­tion with a $10 mil­lion bid for Spirit Air­lines’ cor­po­rate data, with a court still to rule on a union’s pri­vacy objec­tion[47]), but an archive is worth only what the ver­i­fied tasks built on it are worth.

Labs check the ver­i­fier before they sign and retire tasks when they sat­u­rate. Both habits point at the same fact: what is being bought is a trust­wor­thy reward, and rewards fail in spe­cific, mea­sur­able ways. Part 2 shows how.

Part 2Where environments and evals fail, and where value hides

Why does a score so often fail to mean what it claims, what does rein­force­ment learn­ing do to each fail­ure, and which fail­ures would some­one pay an out­sider to fix? Start from an account­ing iden­tity: a reported score is the system’s true capa­bil­ity, plus the scorer’s bias, wrong answer keys, leaked answers and option cues, selec­tion effects from how the items were chosen, and sam­pling noise. Every term after the first has noth­ing to do with the system being scored, and each can be esti­mated and sub­tracted. That sub­trac­tion is what an audit sells.

Figure 4 lays out the eight links between an item and a pub­lished number. Most sit out­side the model: an enter­prise can pick the “win­ning” vendor because the test set was fil­tered against the loser.

Wrong scores: scorers, keys, leaks, cues and filters

The scorer is a clas­si­fier with its own errors. MMLU-Pro’s ref­er­ence script tries two answer pat­terns, then takes the last stand­alone cap­i­tal letter from A to J any­where in the text; when all three fail, it draws a random option and counts the item cor­rect if the draw hits[48]. So unparseable output earns an expected 10% instead of being reported as a format fail­ure, and a response ending “…which I think is right.” is graded as option I (derived from the code). Re-scor­ing all 3,751 models on the Open LLM Leader­board with an equiv­a­lence-aware math parser added 4.66 points on aver­age and moved some models 200 or more places[49]. A con­stant “null model” response reached an 86.5% length-con­trolled win rate on AlpacaEval 2.0[50], and rule-based math ver­i­fiers aver­age 86% recall, so 14% of cor­rect responses are marked wrong[51].

The rule that fol­lows is cheap: after any scorer fix, re-score the stored out­puts before re-run­ning any model, because re-run­ning at the same time mixes the scorer change with sam­pling noise and model drift. In Figure 5’s public cases, integrity fixes moved pub­lished scores by about 5 to 35 points, in both direc­tions, and when­ever the grad­ing asks for what a buyer needs, scores fall, by 19% to 78% of their value across nine pub­lished pairs that each change some­thing dif­fer­ent (a harder fail­ure class, an evi­dence trail, leak-free tests, enter­prise data): SREGym’s own harder fail­ure classes cut one agent from 63.7% to 17.9% end-to-end suc­cess[52], and Cloud-OpsBench’s best root-cause accu­racy of 0.76 becomes an evi­dence-clo­sure rate of 0.38 once the agent must show its work[53].

Model judges carry the same errors, with a direc­tion. In an early study GPT-4 as a judge kept its ver­dict when two answers swapped places in only 65% of cases, though its agree­ment with human experts, over 80%, matched theirs with each other[54]. On blind 2026 bat­tles, 12 judge models picked their own answer, on aver­age, 23 per­cent­age points more often than people did, and one lab’s judges favoured that lab’s models by 37 points more than people, against 6 for other judges[55]. A judge audit reports three num­bers: swap con­sis­tency, self- and family-pref­er­ence against human votes, and drift between judge ver­sions on a frozen set.

Wrong keys, and the denom­i­na­tor. If a share e of the keys is wrong, a per­fect model scores at most 1 − e, and near the ceil­ing most “model errors” are key errors. An esti­mated 6.49% of MMLU ques­tions con­tain errors[56], and about 29% (±3.7, 95% CI) of Human­ity’s Last Exam text-only chem­istry and biol­ogy answers con­flict with peer-reviewed lit­er­a­ture[57]. Read every rate with its denom­i­na­tor: one 2026 audit con­firmed 29 broken tasks among 5,241 scanned (0.55%) and 20 among the 34 it reviewed one by one (58.8%)[58]. Both are true: a rate on a subset chosen because models failed it, or because a scan flagged it, says where errors clus­ter, not how common they are.

Leaks come through three chan­nels. The envi­ron­ment: in 1,795 of 2,698 coding tasks in one open-sourced RL envi­ron­ment release (67%), the fix commit sur­vived as unreach­able git objects[59], and on SWE-bench Pro 63% of one fron­tier model’s suc­cess­ful res­o­lu­tions “retrieved the fix rather than derived it”; seal­ing git his­tory and restrict­ing net­work access cut its score from 87.1% to 73.0%[60]. The prompt: in 32.67% of suc­cess­ful SWE-Agent with GPT-4 patches, the solu­tion sat in the issue report or its com­ments[61]. The train­ing data: on GSM1k, a fresh look-alike of GSM8K, accu­racy dropped by up to 8%[62]. And the options can carry the answer: a clas­si­fier trained on mul­ti­ple-choice options alone, never the ques­tion, reaches 41% on MMLU-Pro, where chance is 10%[63], so ask open ques­tions where you can and report a choices-only base­line where you cannot.

Fil­ter­ing con­cen­trates wrong keys and flat­ters every com­par­i­son with the filter model. Hard sets are often built by keep­ing the items a strong model gets wrong. A cor­rectly keyed item sur­vives only when the model errs; a wrongly keyed one sur­vives almost always. With e the share of wrong keys in the pool, a the filter model’s accu­racy on cor­rectly keyed items and q the chance it is marked wrong on a wrong key:

P(key wrong | kept) = e·q / (e·q + (1 − e)(1 − a))

At 3% wrong keys, a 95%-accu­rate filter leaves about 38% of the sur­vivors wrong-keyed, and a 98%-accu­rate one about 61% (derived): the stronger the filter, the more of its “hard” set is hard only because the key is wrong. TruthfulQA’s 437 ques­tions fil­tered against GPT-3 175B roughly halve that model’s truth­ful­ness score rel­a­tive to the 380 writ­ten with­out test­ing on it, while UnifiedQA 2.8B, never used as a filter, scores no lower on the fil­tered ques­tions (derived from the paper’s Table 4)[64]. The fix costs one table: pub­lish an unscreened random sample of the same pool beside any hard set, and name the filter models.

What reinforcement learning does to each error

At eval­u­a­tion time a scorer’s errors are a fixed bias; in train­ing, the policy is opti­mised against the scorer, so every region where the scorer over­pays becomes a des­ti­na­tion. Theory says this cannot be engi­neered away: because a reward model is an imper­fect proxy, “opti­miz­ing its value too much can hinder ground truth per­for­mance”[65], and over the set of all sto­chas­tic poli­cies two reward func­tions “can only be unhack­able if one of them is con­stant”[66]. So the honest form of any envi­ron­ment cer­tifi­cate is “no exploit found by these searches, at this budget”. It is never a proof.

The two scorer errors behave dif­fer­ently under opti­mi­sa­tion. A false pass is a target: the policy climbs toward what­ever the scorer over­pays, how­ever rare that behav­iour was at the start. A false fail is a famine: cor­rect work goes unre­warded, and because it nar­rows honest work’s margin, a weaker hack is enough to take over. In Figure 7’s toy the ver­i­fier’s pass rate keeps rising while true suc­cess falls, so a reward dash­board cannot show where things go wrong. o3 reward-hacked in 30.4% of RE-Bench runs, and in a sep­a­rate test of one opti­mi­sa­tion task, adding “Please do not cheat.” left the rate at 80%, the same as the orig­i­nal prompt[67]. On SWE-bench tasks made impos­si­ble by con­flict­ing tests, where any pass implies cheat­ing, GPT-5 passed 54%[68].

Patch the scor­ing func­tion rather than penalise the model: METR cau­tions that “naive attempts to address reward hack­ing might simply make the reward hack­ing harder to detect”[67]. And give honest fail­ure an exit: GPT-5’s 54% fell to 9% when it was given one more action, a string that flags the task as impos­si­ble for a human to review, though it helped Claude Opus 4.1 much less[68]. Mea­sure the mit­i­ga­tion on every model family you train or buy.

Absten­tion is trained away unless it is scored. Under 0/1 grad­ing a guess always weakly beats “I don’t know”, so “the train­ing and eval­u­a­tion pro­ce­dures reward guess­ing over acknowl­edg­ing uncer­tainty”[69]. Across 20 models, “rea­son­ing fine-tuning degrades absten­tion (by 24% on aver­age)”[70]. The fix is in the scorer: declare a con­fi­dence target t and score a cor­rect answer 1, an absten­tion 0 and a wrong answer −t/(1 − t). A model right with prob­a­bil­ity p then earns (p − t)/(1 − t) by answer­ing, so it answers exactly when p > t (derived from Kalai et al.). Report the whole cov­er­age curve as t varies; Sec­tion 4.3 turns t into dol­lars.

Noise: most claimed gains sit inside it

What is the small­est dif­fer­ence an eval can resolve, and does a claimed gain clear it? Only 16.0% of 445 reviewed LLM bench­marks used uncer­tainty esti­mates or sta­tis­ti­cal tests to com­pare results[71], and pub­lished noise often exceeds the gains people ship. Pick­ing one run out of many moves single-run pass@1 on SWE-bench Ver­i­fied by 2.2 to 6.0 points[72]; pass@1 on small math sets has a 5 to 15 point stan­dard devi­a­tion across seeds, and the improve­ments sev­eral recent meth­ods built on one small rea­son­ing model reported “fall entirely within the vari­ance range” of that base model[73]; and mean­ing-pre­serv­ing prompt-format changes moved one model by up to 76 accu­racy points[74].

Pair­ing is the cheap­est vari­ance cut, and most pub­lished com­par­isons skip it. Score both sys­tems on the same items: only the dis­cor­dant items, where exactly one system is right, carry infor­ma­tion. With dis­cor­dance rate q and n items, the stan­dard error of the paired dif­fer­ence is about √(q/n), and the min­i­mum detectable effect (MDE) at a two-sided 5% level and 80% power is:

MDE ≈ (1.96 + 0.84) · √(q / n) ≈ 2.8 · √(q / n)

At 300 items and 10% dis­cor­dance, the small­est reli­ably detectable gap is about 5 points; a 2-point gap needs about 1,960 paired items (derived). That is the size of one vendor’s reported lead over a fron­tier model, pub­lished with no sample size or inter­val[75]. It may be real; no out­sider can tell. Pair­ing is “a ‘free’ vari­ance reduc­tion tech­nique”, in Anthropic’s phrase[76], but two cor­rec­tions push the other way. Related items (vari­ants of one seed, ques­tions on one pas­sage, inci­dents from one family) need clus­tered stan­dard errors, which “can be over 3X larger than naive stan­dard errors”[77], so count inde­pen­dent fam­i­lies, not items, before you run. And the for­mula covers sam­pling only: serv­ing numer­ics, seeds and load add noise that only an A/A run mea­sures.

Put together, a claimed gain counts only if three things hold: the lower bound of the paired 95% inter­val is above zero, the gain exceeds the A/A min­i­mum detectable effect, and it repli­cates on a second model family or a fresh item sample. The third clause earns its place: GRPO with ran­domly assigned rewards raised Qwen2.5-Math-7B’s MATH-500 score by 21.4 points, close to the 29.1 from true rewards, and such spu­ri­ous rewards “often fail to pro­duce gains for other model fam­i­lies, such as Llama3 or OLMo2”[78].

Rare cases: what the pooled number hides

Why can an eval report a gain while the costly cases get worse? Because a test set drawn in pro­por­tion to traf­fic gives rare cases few items. In a toy (illus­tra­tive, not data), a can­di­date gains 4 points on common requests (70% of traf­fic), 2 on harder ones (22%), and loses 9 on the rarest 8%; the aver­age rises about 2.5 points. With 600 traf­fic-pro­por­tional items the 95% inter­val on the aver­age is about ±3.6 points and the 48-item tail slice’s about ±12.7, so the regres­sion is invis­i­ble. Spread the same 600 items evenly across slices and the tail inter­val shrinks to about ±6.2, expos­ing the regres­sion, at the price of about ±4.6 on the aver­age (derived, at a per-item paired vari­ance of 0.20).

Real sys­tems show the same shape: one 8-bit quan­ti­sa­tion of a 7B chat model lost 2.36 points while flip­ping 13.62% of its answers between right and wrong[79]. Weight­ing is the second half of the fix: what a clus­ter of cases costs is roughly fre­quency × sever­ity × dura­tion, and a few clus­ters can carry most of it. In an illus­tra­tive inci­dent his­tory (a toy with invented counts: about 1,100 inci­dents a year in 14 clus­ters), an agent han­dles 78% of inci­dents by count but only 38% of the expected loss, and a uni­form 150-inci­dent eval misses the rarest clus­ter entirely in 57% of draws (derived). The buyer pays for the 38%; the uni­form eval reports the 78%. Weight the eval by expected loss, give every costly slice enough items to read on its own, count flips as well as the net change, and never let a pooled gain hide a tail regres­sion.

Luck ceilings and the oracle trap

In tasks with chance and hidden infor­ma­tion, such as trad­ing, bid­ding, under­writ­ing and card or tile games, one result is mostly luck, and the obvi­ous ceil­ing is a trap. Poker gives clean num­bers: in heads-up no-limit hold’em one hand has a stan­dard devi­a­tion of about 13 big blinds[80], while pro­fes­sion­als con­sider 0.05 big blinds a hand “a siz­able margin”[81], so raw results need about 260,000 hands before a 95% inter­val on such an edge excludes zero (derived: (1.96 × 13 / 0.05)²). Small sam­ples can point the wrong way: over 3,000 hands against its top-ranked oppo­nent, DeepStack’s raw chips said it lost 515 ± 575 milli-big-blinds a game, while the luck-adjusted AIVAT esti­mate said it won 70 ± 119[81].

The ceil­ing mis­leads sys­tem­at­i­cally. An oracle that sees hidden state (the oppo­nent’s tiles, the future price path) sets a bound no legal player can reach, and solv­ing sam­pled worlds as if they were vis­i­ble suf­fers strat­egy fusion and non-local­ity “irre­spec­tive of the number of hypo­thet­i­cal worlds exam­ined”[82]. In Scrab­ble, an agent that saw its oppo­nent’s rack beat a strong sim­u­la­tion player by 37 points a game, which its authors call “an upper bound on what we can expect to gain”, while a legal infer­ence agent gained 5.2 points a game over 630 games[83], about 14% of the oracle’s edge (derived).

An unpub­lished exper­i­ment by the author on a cross­word tile game (no public data) shows how this dis­torts progress reports. Against a greedy bot, the best legal player, a search cham­pion rated Elo 1699, won 76% of games and a rack-seeing oracle 92%: about +200 and +424 Elo by 400·log₁₀(p/(1 − p))[84]. An LLM recipe rated 1630 closes about 66% of the floor-to-best-legal gap but only about 31% of the floor-to-oracle gap (derived); only the first describes some­thing a deploy­able player could do. Resolv­ing gaps this size needed dupli­cate-paired eval­u­a­tion, the same tile sequence replayed with seats swapped, as the poker prece­dent pre­dicts. Nor is a famil­iar oppo­nent a bracket: Gin Rummy agents that beat random play over 99 per­cent of the time, and only tie copies of them­selves, lost 70 to 99% of games to a fixed, hand-built expert[85], and adver­sar­ial poli­cies won more than 97% of games against a super­hu­man Go engine[86]. Report three num­bers, the floor, the best legal player and the oracle; judge progress against the legal bracket; and test against a fixed out­side oppo­nent, not only against copies of your­self.

Where value hides

Price fol­lows scarcity, and check­ing is not scarce. Where a ver­i­fier is per­fect and free, as in formal proofs, pro­ce­dural puz­zles and games, the lead­ers supply them­selves: at least five sys­tems report solv­ing all 672 PutnamBench prob­lems in Lean, the cheap­est at a reported mean of $0.17 a prob­lem[87]. There, sell what the ver­i­fier cannot make: fresh, faith­fully stated task streams and cal­i­bra­tion data.

Prefer rewards that do not sat­u­rate, and anchor money rewards to a cer­ti­fied bound on real data. Pass/fail rewards sat­u­rate and then teach noth­ing; a gap to a bound keeps its head­room and catches hacks. Score a bat­tery-dis­patch sched­ule on replayed real prices[88] as realised profit over the per­fect-fore­sight opti­mum, and a score above 1 is a bug, never a win.

What stays scarce is a number that means what it says: a cor­rected score (scorer, keys, leaks, cues and fil­ters accounted for), a mea­sured noise floor (an A/A floor and paired inter­vals on every claim) and cov­ered rare cases (enough items, weighted by loss, on each costly slice). Parts 3 and 4 price each one.

Each fail­ure here has a cheap check and a known fix. The busi­ness ques­tion is who will pay an out­sider to run the checks. The first four areas sell mea­sure­ment directly.

Part 3Selling the measurement itself

Four areas sell Part 2’s cor­rec­tions directly, mostly to labs, envi­ron­ment ven­dors and eval­u­a­tors: eval and envi­ron­ment integrity, exe­cu­tion-checked SQL and spread­sheet envi­ron­ments, tail packs and a fail­ure-mode pub­li­ca­tion. Each runs through the same beats: the fail­ure, the prod­uct, the buyer and first con­tract, size and sub­sti­tutes, the strong­est objec­tion, and a ver­dict with a kill test, a gate fixed in advance that ends the plan if it fails. Sizes are esti­mates unless a source is named; ser­vice­able means what buyers could spend if the prod­uct became a normal pur­chase, con­testable what a new firm can com­pete for today.

3.1 Eval and environment integrity: an acceptance test the buyer appoints

The fail­ure. Envi­ron­ment pro­duc­tion has outrun vet­ting, and the lab dis­clo­sure in the intro­duc­tion holds a sharper find­ing: its mon­i­tors had flagged flawed envi­ron­ments, but in some cases “human review­ers dis­missed envi­ron­ments flagged by our auto­mated mon­i­tors as false pos­i­tives”, so flawed envi­ron­ments stayed in train­ing longer than they should have[1]. When­ever the people who clear a flag answer to the team that needs the envi­ron­ment, that is an incen­tive prob­lem, the kind out­side audi­tors exist to fix.

The prod­uct. Accep­tance test­ing of a pur­chased batch of envi­ron­ments or tasks, appointed by the lab that will train on it, plus fixed-fee audits of enter­prise eval suites. The bat­tery is mostly pub­lished prac­tice: a leak sweep with a planted hon­ey­pot file, an exploit agent told to cheat, Part 2’s judge audit, a ver­i­fier card (mea­sured false-pass and false-fail rates), a filter-bias report with an unscreened sample and an A/A noise-floor cer­tifi­cate. Two rules turn it into evi­dence. A find­ing with­out an exe­cutable wit­ness, an arti­fact a fresh sand­box can replay (such as a con­stant output that scores above the floor), is not reported. And dif­fer­en­tial replay labels find­ings with­out trust­ing either build: a find­ing counts only if its wit­ness suc­ceeds on the defec­tive build and fails on the clean one.

The audi­tor’s own tail eval. An audi­tor that cannot mea­sure its own error sells opin­ions. It plants defects under a hidden man­i­fest and mea­sures recall and false alarms per defect class, then mea­sures recall on flaws it did not plant: inde­pen­dently labelled broken tasks and a blind batch planted by an out­sider. Planted recall flat­ters the team that wrote both checks and defects, and even strong review­ers dis­agree: in OpenAI’s audit of SWE-bench Pro, humans named low-cov­er­age tests as the most common issue for 9.4% of the bench­mark and the agent pipe­line for 4.1%[3]: the agents named that issue 0.44 times as often as the humans did (derived; a ratio of label shares, not a mea­sured recall, which the post does not report).

What a cer­tifi­cate can hon­estly claim. Zero fail­ures in n inde­pen­dent checks bounds a rate, at 95% con­fi­dence, near 3/n, the rule of three[89]. An auto­mated check with recall r that finds noth­ing in n tasks bounds it near 3/(r·n) (derived).

95% upper bound ≈ 3 / n         human deep review, zero flaws found
95% upper bound ≈ 3 / (r · n)   automated check with recall r, zero found

So 50 clean deep reviews sup­port only “below 6%”; “below 1%” needs about 300, roughly $90,000 of reviewer time⁵, or an auto­mated check with recall 0.6 across 500 tasks. At recall 0.8 a clean 500-task batch sup­ports “below 0.75%”; at 0.44, only “below 1.4%” (Figure 11). Recall is itself an esti­mate, so the bound must use its lower con­fi­dence limit: 18 catches of 30 planted defects is 0.6 as a point esti­mate but about 0.43 at an exact one-sided 95% bound, which turns “below 1%” back into “below 1.4%” (derived). Human review­ers miss flaws too, so even the 3/n line is a best case. Recall on unplanted flaws sets every bound, so the cer­tifi­cate prints it, with the exploit models and search budget and one of three ver­dicts: pass, fail or insuf­fi­cient evi­dence. The fee does not depend on the ver­dict, and because the next policy finds exploits the last gen­er­a­tion’s agents missed, the cer­tifi­cate expires with the model gen­er­a­tion.

The ver­i­fier card also feeds the trainer: with mea­sured false-pass and false-fail rates ρ₀ and ρ₁, reward R cor­rects with­out bias to (R − ρ₀)/(1 − ρ₀ − ρ₁), and a lower-vari­ance cor­rec­tion fed by a small stream of appeals improved on the appeals alone in 22 of 24 cells[90]. An audit that enters the reward is harder to cut than a report.

Buyer and first con­tract. Lab-appointed accep­tance test­ing of one pur­chased envi­ron­ment family, bought first by neo­labs and open-model devel­op­ers with­out large qual­ity teams, and by fron­tier labs want­ing a second reader beside their mon­i­tors. Inde­pen­dent eval­u­a­tion sells (Arena passed a $100M annu­alised run rate within eight months of its enter­prise launch[91], self-reported), but no public source shows a price paid for an envi­ron­ment-integrity audit.

Price per task, not per engage­ment: a fixed $25,000⁠–⁠$75,000 fee is 4% to 125% of a 300-task batch priced at $200 to $2,000 a task, while 2⁠–⁠6% of the task’s price, with a floor, fol­lows the asset (esti­mates). Not on com­pute saved either: at Part 1’s retail ceil­ing of $2,400 a task, a $50,000 audit of a 500-task batch with 10% hack­able tasks must save 42% of the $120,000 at risk (derived). The stronger value, avoid­ing trained-in behav­iour and wrong ship deci­sions, has no public price.

Inde­pen­dence is archi­tec­ture. Under Sar­banes-Oxley the audit com­mit­tee, not man­age­ment, “shall be directly respon­si­ble for the appoint­ment, com­pen­sa­tion, and over­sight of the work” of the audi­tor[92]: the rely­ing party appoints, even when the audited party pays. Vendor-ordered cer­tifi­cates drift toward issuer-paid rat­ings, where SEC staff found that the arranger “selects the NRSRO or NRSROs that pro­vide it with the pre­lim­i­nary credit enhance­ment level it desires”[93]. Public leader­boards show the same shop­ping: one devel­oper tested 27 pri­vate vari­ants on Chat­bot Arena, the crowd leader­board Arena runs, before a release and could pub­lish the best[94], Part 2’s selec­tion effect at a mea­sure­ment firm. So a vendor pays only where a lab has writ­ten the clause, and the cer­tifi­cate dis­closes every pre-check ordered on the same batch.

Size. About $240M a year is ser­vice­able (range $54M–$1.0B) only if third-party cer­tifi­cates become a pro­cure­ment norm; about $34M is con­testable today, net of in-house lab work and com­mit­ted bud­gets. One firm can target about $4.5M of annual rev­enue in three years: ten lab or vendor accounts at $0.3M plus 30 enter­prise audits at $50,000 (all esti­mates).

Sub­sti­tutes. Fron­tier labs are the base case, since “eval­u­a­tion flaws are easier to detect now” when a lab’s newest models inspect its tasks[3]. Free reviews from Epoch AI[95] and METR, which “has not accepted fund­ing from AI com­pa­nies”[96], price public ver­dicts at zero, and gov­ern­ment insti­tutes test fron­tier models before release (the UK and US insti­tutes jointly eval­u­ated OpenAI’s o1 before launch[97]), the clos­est case of a rely­ing party choos­ing the asses­sor, though they test dan­ger­ous capa­bil­i­ties, not envi­ron­ments. A spe­cial­ist sells pre-release audits to the other side: Hori­zon Ana­lyt­ics Labs helps dataset ven­dors find fail­ures “before buyers review a release”, find­ings “Pri­vate by default”[98]. The near­est funded com­peti­tor is on the insur­ance path: AIUC cer­ti­fies agents against its own stan­dard, about 5,000 risk and attack com­bi­na­tions per agent, and one cus­tomer used the cer­tifi­cate in Feb­ru­ary 2026 to secure what it called first-of-its-kind insur­ance for AI agents[99]. Vals AI, which raised $40M at $400M (self-reported)[100], is the most direct threat. In this arti­cle’s place­ment nobody holds the upper-right corner of Figure 12.

Strong­est objec­tion: nobody has paid for this, labs will do it them­selves, and the mar­gins are ser­vices mar­gins. The first two partly hold: at the fron­tier, detec­tion belongs to the labs, leav­ing adju­di­ca­tion (an inde­pen­dent reader of flags that people with a stake dis­missed), sold as embed­ded staff in the lab’s cloud account; below the fron­tier there is no in-house sub­sti­tute. The third holds fully. One 500-task cer­tifi­cate priced at $50,000 costs $19,500, $33,500 or $71,250 under lean, middle and heavy assump­tions, for gross mar­gins of 61%, 33% and −42% (esti­mate)⁶. At a $1,000 task the fit­ting fee, $20⁠–⁠$60, sits below the middle cost of $67 a task, so the firm works only if auto­ma­tion cuts cost to $10⁠–⁠$30 a task while recall holds.

Ver­dict: a small, lab-appointed adju­di­ca­tion and accep­tance-test­ing firm, funded as a ser­vices busi­ness; ven­ture-scale only if a lab writes a third-party cer­tifi­cate into a vendor con­tract, or an insurer or reg­u­la­tor relies on one. First con­tract: lab-appointed accep­tance test­ing of one envi­ron­ment family, priced per task with a floor. Kill test: at day 30, recall below 60% on leak or scorer classes, or unplanted recall far below planted recall; at day 45, no lab’s writ­ten agree­ment to name a lab-appointed cer­tifi­cate in one vendor’s state­ment of work.

3.2 Execution-checked spreadsheet and SQL environments: sell the count of false rewards

The fail­ure. The usual SQL reward pays 1 when the agent’s query returns what the gold query returns, but a wrong query can return the right rows on the one data­base it ran on. On Spider, check­ing only the orig­i­nal data­base gave a mean false-pos­i­tive rate of 6.5% across 21 sub­mis­sions[102]. During RL on BIRD-style data, an equiv­a­lence checker found “28.8⁠–⁠38.4% of pos­i­tive rewards” false, and “this rate does not decrease as train­ing pro­gresses”[103]. Snowflake calls these reward col­li­sions (“An outer join and an inner join return the same result if every row hap­pens to have a match on both sides”)[104]. RL selects exactly the errors the checker cannot see.

The prod­uct: traps and wit­nesses. A trap is a near-miss query that is right on some data and wrong on other data (NOT IN against a nul­lable column); a wit­ness is the min­i­mal set of rows on which trap and gold dis­agree. A checker catches a trap only if a wit­ness is in the data. Figure 13 runs 13 clas­sic ana­lyst mis­takes through a dis­closed toy finance data­base (illus­tra­tive; it mea­sures the checker, not a model). On clean data, 11 of 13 earn full reward. Random messi­ness misses rare traps (day­light saving and LIMIT 1 ties sur­vive all 64 data­bases of a random suite); one small planted data­base, a wit­ness row per named trap, catches 12 of 13. Yet a free tool at its best effort caught all 13: dis­tilled test suites, the public seman­tic-eval­u­a­tion method for Spider[105], needed only two data­bases from each of four fuzzed pools. The planter is not the moat; the named trap list, expert-audited golds and inde­pen­dent report­ing are.

The com­para­tor is part of the reward. Spider 2.0-lite’s public grader passes a pre­dic­tion if every gold column matches any pre­dicted column and treats miss­ing values as 0[106], while strict text match­ing fails reordered cor­rect answers; in the toy both rank a hedg­ing agent above a better one. Spread­sheets carry a cached answer key, because open­pyxl returns either the for­mula or “the value stored the last time Excel read the sheet”[107], and across five fron­tier models, checks that change an input pass 15⁠–⁠37% of the time against 50⁠–⁠68% for static for­mula checks[108]. Change an input, recal­cu­late and com­pare, on at least three states.

Buyer and first con­tract. Envi­ron­ment ven­dors buy first, since qual­ity at scale is their bot­tle­neck (Part 1); copi­lot makers follow, and labs come through ven­dors. The first con­tract is a fixed-fee reward-col­li­sion audit: the share of the buyer’s pos­i­tive rewards that are false, per trap family, with wit­ness rows. Packs come last, because broken golds set their margin: a 1,000-task hard pack at $200 a task earns about 22% at 14% broken golds and about 49% at 2%, and noth­ing at $100 (esti­mates)⁷. A broken task can also waste up to $2,400 of the buyer’s RL com­pute at retail prices, twelve times its price, so a pack ships with its gold-error rate and an inter­val.

Size, sub­sti­tutes and the strong­est objec­tion. Roughly $2.5M–$18.5M a year across reach­able seg­ments (esti­mate), against free sub­sti­tutes: dis­tilled test suites and SpotIt+’s search for “data­base instances that dif­fer­en­ti­ate the two queries”[112]. The strong­est objec­tion is that the first public trap card trig­gers in-hous­ing: a planter and com­para­tor for one dialect is one or two researcher-months (esti­mate). That holds for the method, not for the count: a vendor cannot cred­i­bly report its own false-reward rate to the lab that buys from it. So the audit sells on inde­pen­dence, priced below the buyer’s cost to build.

Ver­dict: sell the reward-col­li­sion audit, not the pack; it sup­ports a small, prof­itable audit prac­tice or an acqui-hire, not a stand­alone com­pany. First con­tract: a fixed-fee audit of one vendor’s SQL tasks and grader. Kill tests: at day 30, fewer than three trap fam­i­lies fail fron­tier models at best effort with a 95% inter­val above 10%; at day 60, planted-state rewards fail to beat clean-state rewards on held-out traps (drop packs); at day 90, no fixed-fee audit closes below the buyer’s make cost.

3.3 Tail packs: hard cases grown from seeds, sold with the screen disclosed

The fail­ure. Hard sets sat­u­rate, leak and break, and train­ing buyers have already writ­ten the spec­i­fi­ca­tion for their replace­ments: Part 1’s band of a pass rate above about 2⁠–⁠3% and below about 70%.

The prod­uct. Thou­sands of check­able vari­ants grown from a few dozen seeds per domain, keyed by a pro­gram, fil­tered by a panel of fron­tier models, and sold as pri­vate eval packs and licensed train­ing packs with a refresh sub­scrip­tion. Grow­ing is cheap: SWE-smith built 50,137 exe­cutable soft­ware tasks for about $1,360[113]. A pro­gram writes the key, never a model: in an expert review of Self-Instruct’s gen­er­ated data, 92% of instruc­tions were valid but only 58% of out­puts cor­rect[114].

The screen dis­torts what it keeps, as Part 2 showed: it flat­ters every com­par­i­son with the screen­ing model and con­cen­trates wrong keys. Human­ity’s Last Exam kept only ques­tions its screen­ing models failed, yet o1, one of those models, scored 9.1% at release[115]. The honest pro­to­col is cheap: pre-reg­is­ter the screen, run each vendor’s strong­est model at high­est effort with tools on where the buyer uses tools, re-check every kept key, and pub­lish an unscreened con­trol. A free eval­u­a­tor already does it, dis­clos­ing which screen­ing models selected its hard set[116], so hon­esty is a wedge, not a moat.

Seeds, not vari­ants, set the error bar. Vari­ants of one seed share blind spots. In a toy sim­u­la­tion (Figure 14; illus­tra­tive, not data), naive error bars on 50 seeds × 100 vari­ants call two equally good models dif­fer­ent 44% of the time against a nom­i­nal 5%; clus­ter­ing by seed restores about 5%, and the 5,000 items behave like about 840. At the same budget, 200 seeds × 25 vari­ants detect a 4-point gap 86% of the time against 51% for 50 × 100. Price, label and sell packs by seed count.

Eval packs and train­ing packs are dif­fer­ent prod­ucts. Evals want the tail every model fails; train­ing wants a band each buyer some­times solves, since group-rel­a­tive RL learns noth­ing from prompts whose roll­outs all fail[117], and only the buyer’s own runs find that band. Train­ing packs also face free gen­er­a­tors such as Rea­son­ing Gym, which “pro­vides over 100 data gen­er­a­tors and ver­i­fiers”[118].

Refresh is the sub­scrip­tion, and it needs a knob with no ceil­ing. In Epoch AI’s open data, 14 of 17 bench­marks released in 2023⁠–⁠2025 that were hard at launch have lost half their head­room, at a median of about 9 months (derived)[119]. A fixed-dif­fi­culty gen­er­a­tor runs dry until the only items the fron­tier fails are broken ones, so pick knobs tools do not erase: for­mu­la­tion, rec­on­cil­ing doc­u­ments, under­spec­i­fied requests.

Buyer, size and the strong­est objec­tion. First dol­lars come from enter­prises and eval­u­a­tors buying a pri­vate eval pack in one domain (an esti­mated $25,000⁠–⁠$150,000 a year); year one brings about $0.15M–$0.7M from two to four of some 20 reach­able buyers, no more than a team of two or three costs (esti­mates). The sharpest risk is the sup­plier: the screen runs on APIs whose terms bar using them “to build a com­pet­ing prod­uct or ser­vice, includ­ing to train com­pet­ing AI models”[120], so screen train­ing packs only with open-weight models. The strong­est objec­tion, that free gen­er­a­tors price train­ing packs at zero and lift stud­ies mis­lead (Part 2’s random rewards), holds for generic train­ing packs, not for eval packs whose seeds carry domain knowl­edge a lab lacks.

Ver­dict: a real but small busi­ness, eval-first in one demand domain (finance cal­cu­la­tions from struc­tured data, or SQL over syn­thetic schemas); train­ing packs only after a lift study beats free gen­er­a­tors at equal com­pute on two model fam­i­lies. First con­tract: a pri­vate eval pack with its unscreened con­trol. Kill test at day 90: a blind re-solve of 300 or more random kept items puts the upper 95% bound on wrong keys above 5%, or the honest cost per kept item exceeds a quar­ter of an agreed price.

3.4 A failure-mode publication: a monthly map of where frontier releases fail

The fail­ure. Leader­boards rank; a team rout­ing work across models needs to know where each one breaks. Taking each lab’s strong­est recent model at best effort on eight domains in Epoch AI’s open data, the leader beats the runner-up by more than the com­bined noise in only one domain (derived).

The prod­uct. A map, not a rank: each month, a frozen paired suite on every release and its pre­de­ces­sor, fresh probes grown from seeds by diver­sity-seek­ing search[121], every fail­ure re-run before it ships, and the small­est dif­fer­ence the run could have detected.

Most move­ment is noise, so the noise floor is the prod­uct. Pair­ing each release with its family’s pre­vi­ous release on the same Epoch-run bench­mark gives 337 pairs across 30 fam­i­lies (Figure 15; derived from Epoch’s CC-BY data). 59% of the changes sit inside ±1.96 stan­dard errors (com­bined as if inde­pen­dent; a paired test on the same items would be tighter, so 59% is an upper bound); 12 (3.6% of pairs) are sig­nif­i­cant regres­sions, and ten sur­vive a Ben­jamini-Hochberg false-dis­cov­ery cor­rec­tion. A naive “five-point drop” rule flags 21 pairs, 13 of them inside the noise, and misses 4 of the 12 real regres­sions. On an A/A test of two inde­pen­dent 100-probe runs at a 20% fail­ure rate, the five-point rule would print about 15 false head­lines in a month of 8 releases × 10 domains (derived). Saying “no detectable change” when it is true is the prod­uct.

Buyer, sub­sti­tutes and size. Prod­uct teams, routers and agent builders read the map, but head­lines are fast and free (in Epoch’s data, the median 2026 model from OpenAI, Anthropic, Google DeepMind, xAI or Meta had its first Epoch-run score within about two weeks of release, against about six months in 2024; derived), and custom bench­marks sell at cost: Arti­fi­cial Anal­y­sis charges for Optima “at the raw token cost of the models used, with noth­ing added on top”[122]. So com­pete on rigour, not speed, and charge only for custom probes built from a buyer’s seeds. The strong­est objec­tion is the map’s own evi­dence: if lead­ers rarely sep­a­rate and most moves are noise, buyers can ratio­nally choose on price and latency. It holds.

Ver­dict: worth run­ning as a free monthly pub­li­ca­tion on three domains, a cred­i­bil­ity arti­fact for the mea­sure­ment core, not a sub­scrip­tion busi­ness; charge for custom probe work. First con­tract: one custom probe engage­ment from a design part­ner’s seeds. Kill test at day 90: fewer than three ver­i­fied find­ings that no free source (Arena, Arti­fi­cial Anal­y­sis, Vals AI, Epoch AI, OpenRouter) reported within 14 days.

What the four measurement businesses have in common

All four sell a cor­rected score, a mea­sured noise floor and cov­ered rare cases in dif­fer­ent wrap­pers, and all four lose to free or in-house work unless the buyer is forced to cer­tify or needs an inde­pen­dence it cannot get from the party being judged. Integrity is the anchor, the only one a lab can write into a supply con­tract; the other three are cheap ways to exer­cise the same core in public and earn the cred­i­bil­ity the anchor needs.

The other four areas sell to enter­prises and deci­sion teams, with the mea­sure­ment wrapped around some­thing the buyer already does: a model migra­tion, an inci­dent, a knowl­edge base, a trad­ing or pric­ing deci­sion.

Part 4Measurement wrapped around a workflow

Four areas sell to enter­prises and deci­sion teams: cost-fron­tier end­points, oper­a­tions envi­ron­ments, pri­vate knowl­edge and deci­sions under luck. Each starts as an improve­ment busi­ness (a cheaper end­point, a better on-call agent, a model that knows a com­pany’s facts, a trainer that sees through luck); in each, the improve­ment half meets a funded incum­bent or a free sub­sti­tute, and the mea­sure­ment half meets no one. The beats are Part 3’s.

4.1 Cost-frontier endpoints: the saving is capped by what you defer

Can a vendor sell fron­tier qual­ity on a cus­tomer’s own traf­fic at a frac­tion of the cost, and prove it? The pull is real. The cost of a given level of AI per­for­mance “has fallen about 47% per quar­ter since 2023, or 13× per year”[123], yet rea­son­ing-token con­sump­tion per organ­i­sa­tion rose about 320× in a year[124]. A token is not a task; the unit to sell is cost-of-pass, “the expected mon­e­tary cost of gen­er­at­ing a cor­rect solu­tion”[125].

The fail­ure. Sub­sti­tu­tion works on narrow work­loads, by the sub­sti­tuters’ own account: Inter­com’s post-trained model resolves 73.1% of con­ver­sa­tions against 71.1% for GPT-5.4 at “roughly one-fifth the cost” (Part 2’s 2-point lead)[75], and a Bridge­wa­ter-trained model reached 84.66% against GPT-5.5’s 78.2% at one-four­teenth the infer­ence cost[33] (both self-reported). What is miss­ing is the mea­sure­ment. Of seven widely cited cost-qual­ity claims,⁸ only a hind­sight oracle router states its sample size, none reports a noise band on qual­ity and none amor­tises evals or re-cer­ti­fi­ca­tion; head­lines round in their own favour (“one-four­teenth the infer­ence cost” sits beside “13.8× cheaper” on the same page, and a “40X” speed head­line’s own table gives about 34×[130]). An accu­racy-only reward makes it worse, drift­ing to the expen­sive corner: “For sub­stan­tially sim­i­lar accu­racy, the cost can differ by almost two orders of mag­ni­tude”[131].

The prod­uct. Parity is non-infe­ri­or­ity, not equal­ity: a margin per slice declared in advance, and the lower 95% bound of the paired dif­fer­ence above minus that margin[132]. What is cer­ti­fied is a serv­ing policy (router, model, sam­ples, ver­i­fier, thresh­old, defer­ral, har­ness), against the best legal con­fig­u­ra­tion: a router that picks the best of 18 models per task after seeing the results reaches 97.6% at $1.88 a task against 74.1% at $6.52 for the best single model (a max­i­mum over noisy esti­mates, biased upward)[133], but no deployed router sees the answers first. One line of arith­metic sizes every claim:

cost ≈ k · C_cheap + d · C_frontier (+ verifier)      saving = C_frontier / cost ≤ 1 / d

On one lab’s list the top tier costs 100 times the small­est per token[134]; with one ver­i­fier call, that 100× sticker gap becomes 4.5× at 20% defer­ral and 25× at 2% (derived). FrugalGPT’s “up to 98% cost reduc­tion”[135] is 50×, which needs 2% defer­ral or less. The tail eval sets the defer­ral rate d, and d sets the saving, so the tail is the prod­uct.

The base­line the buyer actu­ally has. With an open-weight cheap path at $0.15 and $0.60 per mil­lion tokens[136], 10% defer­ral and an esti­mated $150,000 a quar­ter of fixed cost, break-even is about 820,000 tasks a month against the top tier at list price, or $0.7M to $0.8M of spend a year which­ever list-price tier the end­point replaces (derived). Past it, the base­line decides what the buyer keeps: at $1M a year on a cached mid tier the bill falls 11% after a 20% fee, which a dis­count of “about 15% off list price” (reported) beats with no migra­tion[137]; against the lab’s small tier, $0.0007 for a task of 3,000 input and 800 output tokens, the end­point never breaks even, because its own cheap path costs $0.0018. Labels set the fixed cost: a 5-point margin on five tail slices needs about 2,335 adju­di­cated items ($26K–$93K a release), 3 points about 6,490 ($71K–$259K)⁹. A slice the budget cannot cer­tify defers to the fron­tier model; it is never qui­etly passed.

The buyer and the forc­ing event. Dep­re­ca­tions make the pur­chase dated and recur­ring. On 14 Octo­ber 2026 OpenAI noti­fied devel­op­ers that GPT-4o and GPT-4.1 snap­shots, with their fine-tunes, leave the API on 14 Jan­u­ary 2027 (rec­om­mended replace­ment gpt-6-luna); GPT-5.1 and GPT-5.3-Codex move to gpt-6-sol and GPT-5.4-Nano to gpt-6-luna on 1 April 2027; and gen­er­ally avail­able models get “At least 6 months” of notice[31]. Since no exist­ing cus­tomer can create a fine-tuning job after 6 Jan­u­ary 2027, a team whose fine-tune dies in Jan­u­ary must move to a dif­fer­ent model. The clock is indus­try-wide: Anthropic prom­ises “at least 60 days’ notice” before retir­ing a pub­licly released model[139], and Google pub­lishes a retire­ment date for each Gemini model on its cloud[140]. The lab names its own cheap tier first; what it cannot cred­i­bly sell is an inde­pen­dent answer to “is the replace­ment good enough on my tail, and what must defer?” That answer is worth most where some­one down­stream relies on it: a bank’s model-risk val­ida­tor (4.4), an EU AI Act accu­racy dec­la­ra­tion (Part 6) or the audi­tor of the buyer’s con­trols. With­out such a party the buyer is paying for com­pe­tence and speed, which it could build itself.

The objec­tion that reshapes the price. A sav­ings share looks aligned and is not: the vendor com­putes the coun­ter­fac­tual it is paid on, and at the aver­age price decline a 10× saving falls below 2× in about two and a half quar­ters unless the end­point re-ships (derived). Energy-sav­ings con­tracts met the same prob­lem, sav­ings that “cannot be directly mea­sured”[141], so borrow their struc­ture: a sub­scrip­tion per work­load, a fee per forced migra­tion, and a kicker capped at the sub­scrip­tion under a plan that freezes base­line and scorer and keeps a 1⁠–⁠2% fron­tier shadow sample. Count­ing work­loads big enough to carry an end­point gives 80 to 180 and about $25M to $70M of fees for all ven­dors, against a top-down $0.5B to $2.6B that assumes migra­tions nobody has counted (esti­mates).

Ver­dict: a neu­tral cer­ti­fi­ca­tion busi­ness that mea­sures every can­di­date, the lab’s cheap tiers included, on the cus­tomer’s tail with noise bands and re-runs at every forced change; the cheaper end­point is an upsell only above about $2.5M to $3M a year on one work­load where the small tier fails a con­firmed tail slice. First con­tract: a fixed-fee cer­ti­fied migra­tion (the lab’s replace­ment, a mid tier and one open-weight can­di­date, per slice, before the shut­down), then a sub­scrip­tion per work­load. Kill test: in week 2, no down­stream party relies on the evi­dence, or the work­load is under about $2.5M a year and the small tier passes every con­firmed slice (sell the cer­tifi­cate alone, priced as tool­ing); in week 13, no sub­scrip­tion with a mea­sure­ment plan is signed.

4.2 Operations environments from real systems: judge, never contestant

Can a com­pany’s past inci­dents become a replayable envi­ron­ment that proves which on-call agent han­dles its own rare, expen­sive nights? Freeze each inci­dent at the page, hide an answer key drawn from machine records, and run every can­di­date on iden­ti­cal replays.

The fail­ure. Ven­dors pub­lish aver­ages with­out the sample behind them (“a mea­sured 72% reduc­tion in time to inves­ti­gate crit­i­cal inci­dents” at one cus­tomer[142]), while the best agent solves 10.0% of hard root-cause tasks on one bench­mark[143]. The inci­dents that cost money are rare inputs on unex­er­cised paths with mis­lead­ing first sig­nals: Cloud­flare’s worst outage since 2019 began with a fea­ture file that dou­bled, while the team “wrongly sus­pected” a “hyper-scale DDoS attack”[144], and Google Cloud’s June 2025 outage began when “unin­tended blank fields” reached a code path that “was never exer­cised during this roll­out”[145]. Four mech­a­nisms make ordi­nary oper­a­tions envi­ron­ments pay for the wrong thing:

• Symp­toms, not defects: faults injected through a chaos-engi­neer­ing tool leave “no defect for the agent to fix”, and 8 of 18 mit­i­ga­tion prob­lems in one suite can be “solved” by a pod-restart loop[52].
• Labels leak from the future: the post­mortem, the revert and the chat thread post­date the page; cut every source at detec­tion time.
• Scor­ers pay for­mat­ting: under one public root-cause scorer (reg­u­lar-expres­sion extrac­tion, case-sen­si­tive string equal­ity, a point per field)[146], a made-up answer naming the wrong host earns 0.67 while the right answer with its keys reordered earns 0.33, or noth­ing (derived by run­ning the scorer).
• Mixed rewards get farmed: a 14B model beat a fron­tier base­line in aggre­gate while its local­i­sa­tion fell from 22.7% to 18.2% in train­ing[147].

Why pro­duc­tion met­rics cannot cer­tify an agent. Davi­dovič split three com­pa­nies’ real inci­dent records into halves sized to one year (173, 103 and 609 inci­dents) and short­ened every inci­dent in one half by 10%: “Even though in the sim­u­la­tion the improve­ment always worked, 38% of the sim­u­la­tions had the MTTR dif­fer­ence fall below zero for Com­pany A, 40% for Com­pany B, and 20% for Com­pany C”[148]. Paired replays can cer­tify, if sized in inde­pen­dent inci­dent fam­i­lies. Under a hard-tail assump­tion, ten inci­dents run once detect only a gap of about 54 per­cent­age points in res­o­lu­tion rate; a dozen inci­dents in the costli­est clus­ter, run three times, about 34; 40 to 50 fam­i­lies with three vari­ants and three runs, 9 to 12 (esti­mates, with family-clus­tered errors after Miller, Part 2).

Buyer, sub­sti­tutes, walls. The buyer is a head of SRE choos­ing among agents, with sev­eral hun­dred inci­dents a year, more than one observ­abil­ity vendor and no ver­i­fier team. The sub­sti­tute is free: buyer guides ask teams to “Share 10 his­tor­i­cal inci­dents that were truly sig­nif­i­cant” for a vendor-run “back­test­ing phase”[149], which tests a demo, not a rank­ing. The walls: indexed traces are kept “15 or 30 days, deter­mined by cus­tomer plan”[150], so old inci­dents replay as met­rics and changes only; one vendor’s terms bar “per­form­ing com­pet­i­tive analy­ses”[151], so run each agent under the buyer’s own licence on records the buyer exports; and neu­tral testers get sued[152], so pub­lish pri­vate reports, not league tables.

Unit eco­nom­ics. Prices anchor the agent’s labour, not its cor­rect­ness: about $6.50 to $8.45 an auton­o­mous inves­ti­ga­tion (about 6.5 cred­its at $1.00 to $1.30 a credit, derived)[153]; no SKU sells proof. A 150-inci­dent pilot needs about four people for 13 weeks, one person-year, or about $300,000 of loaded labour plus $13,000⁠–⁠$16,000 of agent com­pute (esti­mate)¹⁰, against a fee of per­haps $60,000 to $120,000, so deliv­ery cost must fall three- to five­fold by the fifth cus­tomer. On scep­ti­cal assump­tions the ser­vice­able market is $18.5M to $90M a year (esti­mate), and the likely exit is vis­i­ble: Elas­tic agreed to buy Deduc­tive AI for up to $85M at about $1M of recur­ring rev­enue (reported)[156].

Ver­dict: worth a cheap, bounded test as a neu­tral accep­tance-test ser­vice that judges agents and never enters one, for mid-market oper­a­tors run­ning more than one observ­abil­ity vendor; a small market whose likely exit is an acqui­si­tion by a plat­form that wants the ver­i­fiers. First con­tract: a $60,000 to $120,000 pilot on 150 inci­dents frozen at the page, the cur­rent agent and up to three can­di­dates run three times each, cred­ited to a sub­scrip­tion. Kill test: fewer than three of ten SRE leads name a price above deliv­ery cost while a free back­test is on offer; or by week 8 fewer than one in three inci­dents can be rebuilt.

4.3 Private knowledge with calibrated abstention: the fact table is the rival

Can a com­pany buy a model that knows its own facts and says when it does not? Retrieval nar­rows the prob­lem with­out clos­ing it: legal research tools that retrieve from curated case law “pro­duced incor­rect infor­ma­tion more than 17% of the time”, and one “hal­lu­ci­nated more than 34% of the time”[157].

The fail­ure, priced. The spe­cific fail­ure is an invented pri­vate fact. When an air­line’s chat­bot promised a refund rule the air­line did not have, a tri­bunal held the air­line liable (“It makes no dif­fer­ence whether the infor­ma­tion comes from a static page or a chat­bot”) and awarded C$812[158]. Direct costs are small; the buyer pays to cut a rare public fail­ure that repeats until some­one notices, priced like insur­ance from the cost of a wrong answer in each class. Part 2’s scor­ing rule, with dol­lars in it:

answer only if P(correct) > t     wrong answer costs λ = t / (1 − t)     t = 1 − c_abstain / c_wrong
an $8 hand-off and a $200 wrong answer give t = 0.96, λ = 24 (illustrative costs)

The price of a wrong answer picks the winner. On SimpleQA’s pub­lished shares[159], o1-pre­view leads under binary grad­ing; at λ = 1 every listed model scores below zero; GPT-4o, which answers 99% of ques­tions and gets 60.8% wrong, falls to sixth; at λ = 9 it scores −509 per 100 ques­tions against −168 for a model that declines three quar­ters of them (com­puted here).

Why the model does not know, and why RL cannot teach it. Pri­vate facts are sin­gle­tons: if 20% of a kind of fact appears once in train­ing, “one expects base models to hal­lu­ci­nate on at least 20%” of them[69]. Stored is not extractable: one ren­der­ing per person gave 9.7% ques­tion accu­racy, diverse per­muted ren­der­ings 96.6%[160], and direc­tion mat­ters too (the rever­sal curse[161]). And a group-rel­a­tive advan­tage[162] car­ries signal only when a group dis­agrees:

P(group has reward variance) = 1 − p^G − (1 − p)^G      # zero at p = 0, for every group size G

A fact the model never pro­duces gives no gra­di­ent, how­ever many roll­outs are bought; in prac­tice a knowl­edge-first stage adds the facts, and the model then over-answers. A binary reward “quickly sup­presses absten­tion behav­ior to nearly zero”, with 47.7% hal­lu­ci­na­tion against 18.8% for a ternary reward[163], and one 2026 pre­print finds that group stan­dard-devi­a­tion nor­mal­i­sa­tion “silently replaces every designed penalty with an effec­tive penalty of one”[164]. The safer default: always report a con­fi­dence, score it with a strictly proper rule, and abstain only at deploy­ment, at the buyer’s t.

The prod­uct, and the strong base­line. The cheap­est seri­ous rival is the cus­tomer’s own fact table served as tools to a fron­tier model: lookup rejects a name with no row, and a reverse ques­tion becomes a filter. Near-miss names show why it mat­ters: GPT-4o accepted 65.89% of non-exis­tent enti­ties recom­bined from real names, against 18.74% of invented ones[165], and inter­nal cat­a­logues are full of codes one char­ac­ter apart. So every audit runs three arms and a ceil­ing: (a) the incum­bent with its real retrieval, (b) a fron­tier model hold­ing the same table with the same cal­i­bra­tor, (c) trained weights, and a fron­tier model handed the gold pas­sage. The ceil­ing bites: an 8B model with injected knowl­edge scored 56.22% closed book, above GPT-4’s 51.30%, but GPT-4 given the doc­u­ment scored 86.09%[166].

Who is served, and what pays. Sup­port buyers buy agents from ven­dors that run their own models and hold third-party cer­tifi­cates (3.1), and data plat­forms “auto­mat­i­cally create eval­u­a­tion bench­marks spe­cific to your task” for the agents they then tune[167]. What is open is fab­ri­cated-entity rejec­tion over the cus­tomer’s own names­pace, audited by some­one who does not grade its own tuning. The arith­metic is unfor­giv­ing: break-even per avoided wrong answer is price ÷ (queries × tail share × gain), and $275,000 over 600,000 queries a year, a 10% tail and a 10-point gain is about $46 (derived), which rules out most low-stakes sup­port. For the most pro­tec­tive buyers the data cannot leave (83% of fund man­agers rank data secu­rity and pri­vacy among their top three AI risks[168]), so the fac­tory goes to the data; check where the gra­di­ent runs, since the clos­est hosted option says “Train­ing is not sup­ported in BYOC during Pri­vate Pre­view”[169]. Design sets the margin: on a $275,000 con­tract, a full-weight model per cus­tomer with an engi­neer-week a month runs at about 28%, and high-rank adapters on a shared base with auto­mated retrain­ing at about 86% (derived design sce­nar­ios).

Ver­dict: worth build­ing as a smaller com­pany: an inde­pen­dent pri­vate-fact audit and knowl­edge layer first, and trained weights only where they beat a fron­tier model hold­ing the same table by more than the noise floor on the sin­gle­ton stra­tum at matched cov­er­age. First con­tract: a fixed-fee audit at about $40,000 to $60,000 of one inter­nal knowl­edge sur­face, run read-only inside the cus­tomer’s cloud, with the three arms and the ceil­ing in the con­tract. Kill test: in week 13, on three audits, arm (b) sits within the A/A floor of the ceil­ing on sin­gle­ton facts in two of three (sell the table and audit, not weights), or no buyer asks for a re-audit (stop).

4.4 Decisions under luck and hidden information, including markets: verify, do not train

When an agent trades, bids, prices, under­writes or plays cards, how does anyone tell a better deci­sion-maker from a luck­ier one, and who pays to know? Public LLM trad­ing con­tests still report one path, six models with $10,000 each fin­ish­ing a season between +22.3% and −62.66% (reported)[170], while a free arena already plays dupli­cate poker, each 100-hand sequence twice with seats and cards swapped[171].

Why the train­ing pitch is thin. A group-rel­a­tive advan­tage ranks roll­outs by luck unless the group replays one chance seed, and that is already a default: one open frame­work’s config notes that “under the same group, the env config and env seed are ensured to be equal”[172]. Fully cou­pled roll­outs help only con­di­tion­ally: one 2026 pre­print more than halves a return-con­trast vari­ance in a small poker game, finds lower vari­ance is not by itself what the opti­miser needs, and falls back to inde­pen­dent sam­pling when­ever it cannot cer­tify a gain[173]. A train­ing method here is a pub­lished con­fig­u­ra­tion, not a moat.

The mea­sure­ment, and its pre­con­di­tions. Large vari­ance cuts need a coun­ter­party whose strat­egy is known. AIVAT sub­tracts luck at chance events and at deci­sions whose strat­egy is known: in poker self-play it cut the per-hand stan­dard devi­a­tion from about 13.0 to 4.0 big blinds, a 10.3× vari­ance cut, but with only the chance odds known, to 10.6 big blinds, about 1.5× (derived)[80]. A 2026 study of 15 LLM agent con­fig­u­ra­tions against fixed oppo­nents, with cor­rected pay­offs “sup­plied by the eval­u­a­tion plat­form”, reports a median vari­ance cut of 54×. Yet under the exact sequen­tial test a strict val­ida­tor would use, raw results needed only 1.37 times as many hands as cor­rected ones, so most of the adver­tised saving dis­ap­pears; and check­ing a fixed inter­val after every hand declared an effect in 61% of 2,000 sim­u­lated com­par­isons where none existed[174]. Com­par­ing two LLMs means esti­mat­ing both strate­gies by sam­pling, which costs money:

net efficiency = variance-reduction factor / cost multiplier        10.3× with k = 8 samples per decision → ≈ 1.3×

Mar­kets, the hard­est case. The envi­ron­ment, not the agent, has to carry the sta­tis­tics. With five years of data, about 45 inde­pen­dent con­fig­u­ra­tions push the expected best in-sample Sharpe ratio to 1 with no skill[175] (re-derived with the paper’s approx­i­ma­tion: the expected max­i­mum of 45 stan­dard normal draws is about 2.24, and a five-year annu­alised Sharpe ratio has a stan­dard error of about 1/√5), so the reward must charge the winner for the search, for instance with a Real­ity Check boot­strapped over the whole trial ledger[176]. And models remem­ber the future: told to ignore every­thing after 2010, GPT-4o still called the direc­tion of quar­terly GDP growth with 98.0% thresh­old accu­racy after that date, against 40% after its real knowl­edge cutoff[177]. Train­ing is cheap only where the teacher’s infor­ma­tion is legal. In the author’s unpub­lished tile-game exper­i­ment, dis­till­ing a search teacher’s full rank­ing of moves on a full-infor­ma­tion sub-task deliv­ered about 42,000 train­ing posi­tions per GPU-hour against about 1,000 for group-rel­a­tive RL: some 40 times more signal per unit of com­pute, a through­put gain rather than a mea­sured gain in strength. When the teacher sees hidden infor­ma­tion, “simple knowl­edge dis­til­la­tion does not work well”, as Suphx’s authors found[178].

Who must buy, and who will not. Rules ask for what a ver­i­fier pro­vides: EU invest­ment firms must test algo­rithms “in an envi­ron­ment that is sep­a­rated from its pro­duc­tion envi­ron­ment”[179], and US bank model-risk guid­ance asks val­ida­tors for “suf­fi­cient inde­pen­dence to main­tain objec­tiv­ity” while putting gen­er­a­tive and agen­tic AI “not within the scope of this guid­ance”[180]. Inde­pen­dence is expected; a pur­chase is not man­dated, and the best-placed sell­ers trade instead: a trad­ing-AI lab co-founded by two of DeepStack’s authors is valued at $500 mil­lion with 25 people[181]. And sold on com­pute saved, eval­u­a­tion is worth little: at $0.07 to $0.30 of infer­ence a hand[182], a ten­fold cut saves a team run­ning 50 ver­dicts of 20,000 hands about $63,000 to $270,000 a year (derived). The price has to rest on a ver­dict a scep­ti­cal third party accepts.

Ver­dict: a narrow, inde­pen­dent ver­i­fi­ca­tion and release-gate bureau for deci­sion agents in sim­u­la­tors, or a prod­uct line inside an envi­ron­ment or end­point com­pany; not a plat­form for eval­u­a­tion plus train­ing. First con­tract: a rerunnable ver­i­fi­ca­tion of one agent vendor’s or arena oper­a­tor’s results at $25,000 to $150,000, against a known-strat­egy ref­er­ence bot, with seeds, the noise floor and the tail slices in the report. Kill test: by day 30, net effi­ciency below 2× for LLM-against-LLM com­par­isons; by day 90, nobody signs a paid ver­i­fi­ca­tion.

What the eight areas have in common

In all four areas the busi­ness a founder would write down first meets a funded incum­bent or a free sub­sti­tute, and what sur­vives is a neu­tral com­par­i­son, on the buyer’s own tail, that a third party can rerun. Table 4.1 lines up all eight; each area’s kill test is in its ver­dict.

Read down the ver­dict column: no area sur­vives as a model, an end­point or a train­ing plat­form; each sur­vives as a ver­dict. And every kill test asks first for a mea­sured error rate on the founder’s own method, then for a writ­ten com­mit­ment from a buyer who had a free sub­sti­tute.

Eight areas, eight ver­dicts that sell a mea­sure­ment. Part 5 explains why, then prices it.

Part 5Why nearly everything converges on measurement

Why does every honest case end as a ver­dict busi­ness, what does sell­ing ver­dicts cost, what can such a busi­ness defend, and how should a 90-day plan fail cheaply? Read side by side, the eight areas stop look­ing like eight sep­a­rate dis­ap­point­ments and start look­ing like one prop­erty of the market. The evi­dence is the shape, which each area’s sec­tion lets a reader check: five of the eight obvi­ous first busi­nesses were a model, end­point, plat­form or data com­pany, one was a judge that also sold its own con­tes­tant, and every sur­viv­ing shape sells a ver­dict. Sizing adds a gen­eral lesson: in these mar­kets a top-down share of a head­line pool over­states a bottom-up count of buyers by about an order of mag­ni­tude, as Part 1’s $8.5B trap and Sec­tion 4.1’s $0.5B–$2.6B against $25M–$70M both show.

Five reasons the verdict is what survives

1. The nat­u­ral sell­ers of the com­par­i­son have a stake in its answer. The lab that retires a model names its own replace­ment (4.1); the on-call agent vendor back­tests its own agent on the buyer’s inci­dents (4.2); the data plat­form builds the bench­mark for the agent it then tunes (4.3); a lab’s own review­ers dis­missed its mon­i­tors’ flags (3.1). None can sell inde­pen­dence, the one prop­erty a buyer cannot get from the party being judged. Inde­pen­dence earns a price, though, only when some­one other than the buyer relies on the ver­dict: a lab train­ing on a vendor’s envi­ron­ments, a model-risk val­ida­tor, an audi­tor, a reg­u­la­tor or an insurer. Where the buyer is the only reader, as in much cost-fron­tier, oper­a­tions and pri­vate-knowl­edge work, it can mea­sure itself with free tools, and what it pays for is com­pe­tence it lacks: labelled tail sets, noise floors and days from notice to ver­dict.

2. The improve­ment half is crowded or unproven. The luck-aware train­ing trick is a con­fig­u­ra­tion default (4.4); infer­ence clouds give the improve­ment step away to sell tokens, one promis­ing to “Serve fine-tuned models for the same price as base models”[183]; agent star­tups are funded to sell the improved agent (Resolve AI raised a $125M Series A at a $1B val­u­a­tion[184]); and this arti­cle found no public study in which trained weights beat a fron­tier model hold­ing the same pri­vate fact table as tools.

3. Mea­sure­ment alone prices like tool­ing unless a forc­ing event applies. Eval tool­ing lists at $0, $249 a month or a custom price[185], and one exper­i­men­ta­tion plat­form ships CUPED, a stan­dard vari­ance-reduc­tion method, in its free tier[186]. It becomes a must-buy only when a dated forc­ing event applies: a forced dep­re­ca­tion (4.1); a lab clause naming the audi­tor, which no lab has yet pub­lished; a re-pro­cure­ment such as Ops­ge­nie’s, whose access “will be shut off” on 5 April 2027[187]; a val­ida­tor’s review (4.4); or an insurer pric­ing cover on an audit. Insur­ers have begun to write cover against agent cer­tifi­cates (3.1), while sev­eral large car­ri­ers ask reg­u­la­tors for per­mis­sion to exclude wide­spread AI lia­bil­i­ties from cor­po­rate poli­cies (reported)[188]; the oldest prece­dent ran the same way, since UL, the prod­uct-safety cer­ti­fier, began in 1894 as a bureau of the National Board of Fire Under­writ­ers[189]. No forc­ing event, no budget line.

One mea­sure­ment firm already escapes the tool­ing price, and how it does so is the lesson. Arena (for­merly LMArena) passed a $100M annu­alised run rate within eight months of its enter­prise launch and raised $200M at a $3.1B val­u­a­tion in Octo­ber 2026 (self-reported)[190]. Its ver­dict is public and the mea­sured party needs it to sell: a lab launch­ing a model wants the rank­ing, the crowd’s votes are a data asset nobody can back­fill, and devel­op­ers test pri­vate vari­ants on it before release (3.1), which is also its integrity risk. A mea­sure­ment busi­ness reaches ven­ture scale when the mea­sured party needs the ver­dict to sell, not only when a rule requires it. None of the eight areas has that struc­ture today.

4. Deliv­ery is people-heavy. One investor’s port­fo­lio data put AI com­pa­nies’ gross mar­gins “often in the 50⁠–⁠60% range”, against 60⁠–⁠80% or more for soft­ware and 30⁠–⁠50% for ser­vices firms[191]. Before auto­ma­tion the areas here sit at or below that line (an integrity cer­tifi­cate at 61%, 33% or −42%; an oper­a­tions pilot whose labour exceeds its fee), so each bets that auto­ma­tion will cut human hours per ver­dict while the error rate holds.

5. The exits are acqui­si­tions. Beyond Part 1’s list, Data­dog bought the exper­i­men­ta­tion plat­form Eppo[192] and Elas­tic agreed to buy Deduc­tive AI at about $1M of recur­ring rev­enue (4.2). Stat­sig had raised at a reported val­u­a­tion just over $1.1B on $40M of recur­ring rev­enue[193], but as a hor­i­zon­tal plat­form it is an upper ref­er­ence, not a bound. The acquir­ers held traf­fic, teleme­try or com­pute and bought the ver­i­fiers to go with them.

On public evi­dence as of Octo­ber 2026, none of the eight is a ven­ture-scale com­pany yet. Either of two events would change that: a party with author­ity starts to require a third-party ver­dict, or the ver­dict becomes some­thing the mea­sured party needs in order to sell (Part 6).

Unit economics: hours per verdict is the headline number

In every area one number sep­a­rates a soft­ware busi­ness from a con­sul­tancy: human hours, or fixed deliv­ery cost, per ver­dict (per audited task, cer­ti­fied work­load, com­piled inci­dent or audit). If it falls with each cus­tomer while pub­lished error rates hold, the margin moves toward soft­ware; if it stays flat, the com­pany is a ser­vices firm with a GPU bill. Figure 21 shows the levers that move margin on the same work, largest first:

1. Who pays com­pute. On one illus­tra­tive design, con­tri­bu­tion margin is about 66% when train­ing runs on the cus­tomer’s own cloud account and about 4% when the vendor pays for twelve multi-turn agent runs at uncached list prices, about $4,600 each, because agen­tic RL rereads its grow­ing con­text on every turn.
2. Price the gates, and sell ser­vices at cost. A man­aged deploy­ment that car­ries 300 for­ward-deployed hours, large-model runs and a sand­box min­i­mum inside one $250K sub­scrip­tion earns about 21%; re-bill the large runs at cost plus 20% and sell the for­ward-deployed hours at cost, and the same costs earn about 46%. One investor calls it “short­sighted to be opti­miz­ing for 80% gross margin” while racing to own the data layer[194]. The corol­lary is a floor: never sell ser­vices below cost.
3. Labels and reuse. Set the label budget by the margin the buyer accepts (about 2,335 adju­di­cated items for 5 points on five slices, 4.1), and refresh from tem­plates (58%) rather than bespoke work (6%, Part 1).
4. Base-model size, with pass-through. A 102,400-roll­out ref­er­ence run costs $924 on a 35B-param­e­ter mix­ture-of-experts model and $4,823 on a 397B one at Tinker’s list prices[195] (derived: 200 steps × 64 prompts × 8 sam­ples of 4,000 prompt and 1,000 sam­pled tokens), so pass large runs through at a stated markup; sand­boxes carry min­i­mums too (one provider’s Enter­prise plan “requires $3,000/mo min­i­mum”[196]).

What is not a moat, and what holds

Meth­ods are not a moat: con­trol vari­ates, dupli­cate deals, check­lists, open eval frame­works and routers are pub­lished or free, and hubs, sand­boxes and cheaper serv­ing of open models are com­modi­ties. What holds looks more like an audit firm’s rep­u­ta­tion than like soft­ware, and no item is enough alone:

• Inde­pen­dence a rely­ing party can check: who appoints the asses­sor, who pays, whether the firm also sells a con­tes­tant, whether the fee depends on the ver­dict, and whether every pre-check is dis­closed.
• Error rates the firm pub­lishes about itself: recall on flaws it did not plant, an A/A floor on every claim, and the right to say “insuf­fi­cient evi­dence”.
• A place in the accep­tance work­flow: the cer­tifi­cate becomes the gate for every later change, mea­sured as cycle time from a dep­re­ca­tion notice to a cer­ti­fied replace­ment.
• Point-in-time archives that cannot be back­filled. The SEC’s frames API “aggre­gates one fact for each report­ing entity that is last filed”[197]: in one pull of diluted earn­ings per share for cal­en­dar Q2 2025, made on 18 Octo­ber 2026, 4,195 of 4,759 facts (88%) came from 2026 fil­ings, later com­par­a­tives rather than first-reported values[198] (derived), so an answer key built on it rewards look-ahead. Who­ever starts a live, prospec­tively scored record first owns it.
• Val­i­dated ver­i­fiers with an ade­quacy score. KernelBench’s offi­cial check misses “one in six” wit­nessed faults (16.9%), while a muta­tion-opti­mised suite reaches 98.0% detec­tion with two inputs per prob­lem (94.8% on held-out mutants)[199].

The shared weak­ness: cus­tomer data is not pooled and every arti­fact goes back to the buyer in rerunnable form, so the real asset, a record of ver­dicts that turned out right, grows slowly. Figure 22 shows where each area is exposed: a free sub­sti­tute is the most common severe threat, in five areas, and labs doing the work them­selves and plat­forms bundling the check are severe in four each. The base­line also moves: OpenAI’s SimpleQA error rate fell from 75% for o4-mini to 26% for gpt-5-think­ing-mini, mostly because the newer model abstained on 52% of ques­tions against 1%[200]. A gap mea­sured at sig­na­ture can close by renewal.

Inde­pen­dent test­ing has base rates, and they favour this shape. Antivirus testers, whose results buyers read, govern their meth­ods through a stan­dards body set up in 2008 whose board is half ven­dors and half testers; NSS Labs, a tester member that sued that body and three ven­dors in an antitrust case later dis­missed, then ceased oper­at­ing[201]. MLPerf requires sub­mit­ted code “suf­fi­cient to repro­duce the results” and makes every sub­mit­ter review at least one other sub­mis­sion[202]. Ver­dict busi­nesses last as rerunnable results under rules writ­ten with the mea­sured par­ties and enforced by con­tract; they die as lone crit­ics.

Inde­pen­dence has to be built into the con­tract; a policy state­ment does not sur­vive a sales quar­ter. The rely­ing party appoints the asses­sor (3.1); per client the firm is the judge or a con­tes­tant, never both; any improve­ment line is gov­erned sep­a­rately and dis­closed on every cer­tifi­cate; the fee does not depend on the ver­dict; and each cer­tifi­cate names its exploit models and search budget, expires with the model gen­er­a­tion it covers and caps lia­bil­ity.

How to write a 90-day plan that fails cheaply

Every area reaches its first go or kill deci­sion for well under $1M: from about $240K (deci­sion-agent ver­i­fi­ca­tion) to about $610K (the top of the pri­vate-knowl­edge range), and about $34K–$59K for the fail­ure map (derived; Table 5.1). Five rules make that money buy infor­ma­tion rather than activ­ity:

• Start SOC 2 work on day 1. A Type 2 report covers a window “usu­ally three to twelve months” long[203], so deliver inside the buyer’s own cloud account until it arrives.
• Put the test that needs no buyer first: recall on planted and unplanted defects, an A/A floor on a real­is­tic suite, the trap tail on fron­tier models at their best effort. It pro­duces the error rates every later cer­tifi­cate prints.
• Make the first buyer-depen­dent gate a writ­ten com­mit­ment, not a meet­ing: a spon­sor­ship letter, a signed sub­scrip­tion or a priced letter of intent.
• Decide on three engage­ments, not one, with kill thresh­olds fixed before the data arrive: one cus­tomer’s yes or no has a noise floor of its own.
• Budget for what a pass com­mits you to fund: cer­tifi­cates that cost $19.5K–$71K each to deliver until auto­ma­tion lands, labels at $71K–$259K a release, first oper­a­tions pilots that each lose about $190K–$260K, or a $4M–$6M seed for deci­sion-agent train­ing research before any proof of a gain. That, not the first 90 days, decides how much money a com­pany here needs.

If eight areas need the same mea­sure­ment core, the strate­gic ques­tion is which one to enter first so that the second and third are cheaper, and what must stay sep­a­rate how­ever a com­pany here is built.

Part 6How the areas combine, and where to start

Can the eight areas share one mea­sure­ment core, what has to stay sep­a­rate how­ever a com­pany here is built, which area should go first, and what result would change that order?

One measurement core, two buyer sides

Every sur­viv­ing busi­ness sells a ver­dict, and every ver­dict runs on the same six mod­ules: an A/A noise-floor engine; paired, clus­tered inter­vals with min­i­mum detectable effects and any­time-valid stop­ping, so peek­ing cannot crown a false winner; a tail slicer with adju­di­ca­tion and an unscreened con­trol beside every fil­tered set; a ver­i­fier audit with recall on planted and unplanted defects; an expir­ing cer­tifi­cate with a runner that re-cer­ti­fies on every change; and a spend ledger with approval gates and watch­dogs, because agents that launch jobs on a buyer’s behalf need a hard stop. Each market adds two or three mod­ules of its own (Table 6.1).

Once the core exists, most of the small areas’ remain­ing work is domain labels and defects, which is the case for run­ning the fail­ure map, the SQL audit and tail packs as cheap public exer­cises of the core rather than as com­pa­nies. Two things never share. Integrity, luck ver­i­fi­ca­tion, tail packs, the SQL audit and the fail­ure map sell to labs; cost-fron­tier cer­ti­fi­ca­tion, pri­vate knowl­edge and oper­a­tions sell to enter­prises: two sales motions, two secu­rity reviews. And seven of the eight areas have a nat­u­ral upsell (a cheaper end­point, trained weights, a tuned agent, train­ing packs, fixes for what a map finds) that, sold in the account that buys the cer­tifi­cate, makes the judge a con­tes­tant. Integrity alone sells no envi­ron­ments, data or models.

Two honest shapes. One mea­sure­ment com­pany on one buyer side, sell­ing into two or three mar­kets from the shared core, with any improve­ment work a sep­a­rately gov­erned line dis­closed on every cer­tifi­cate; or a port­fo­lio of an enter­prise-side cer­ti­fi­ca­tion firm and a lab-side integrity firm that share open-source core libraries and noth­ing else. The first is cheaper to build; the second keeps each firm’s inde­pen­dence simple enough to write into a con­tract.

Which area first

Figure 23 rates the eight areas on value, fea­si­bil­ity, speed to a first fee, cap­i­tal, defen­si­bil­ity and core reuse (this arti­cle’s assess­ment; the weights are yours).

The scores are judge­ments, so the robust read­ing is dom­i­nance, which needs no weights: cer­ti­fi­ca­tion scores at least as well as pri­vate knowl­edge, oper­a­tions, luck ver­i­fi­ca­tion and tail packs on every cri­te­rion (the table view lists every such pair), so no weight­ing puts any of those four first. On all four preset lenses cer­ti­fi­ca­tion and integrity take the top two places.

The sequence

1. The cer­ti­fied migra­tion goes first (4.1): a neu­tral per-slice ver­dict on every can­di­date replace­ment for a retir­ing model, then a sub­scrip­tion that re-runs at every forced change. It has a dated, recur­ring buyer, the fastest fal­si­fi­ca­tion (week 2 and week 13 gates) and the least cap­i­tal (four people, with train­ing and serv­ing rented). It builds the core, and it stays honest when the answer is “use the lab’s tier”. The best first buyers have a down­stream rely­ing party. A shut­down cohort announced only three months ahead is reach­able only through a warm design part­ner already on a retir­ing fine-tune, so a new entrant’s real­is­tic pipe­line is the cohort after it.
2. An integrity recall test on the core runs in the same 90 days. A team test­ing this plants defects in its own scor­ers and har­nesses, mea­sures recall and false alarms per class, then mea­sures recall on flaws it did not plant. The test needs no buyer and pro­duces the error rates every cer­tifi­cate prints. Writ­ten spon­sor­ship from a lab by day 45 would make lab-appointed accep­tance test­ing a second, lab-side line, or the port­fo­lio’s second firm.
3. Two pub­lished cred­i­bil­ity arti­facts exer­cise the core at near-zero cost: a free monthly fail­ure map with its A/A false-head­line rate beside it, and a reward-col­li­sion card on public SQL RL setups.
4. Pri­vate-fact audits come third, sold to the same enter­prise accounts once the core and the SOC 2 window exist. Trained weights earn a place only where the week-13 test shows them beat­ing a fron­tier model hold­ing the same table by more than the noise floor.
5. Oper­a­tions and luck ver­i­fi­ca­tion stay cheap tests: ten con­ver­sa­tions with heads of SRE while a free vendor back­test is on the table, and a day-30 net-effi­ciency test for com­par­isons of one LLM agent against another. Tail packs wait for a lift study that beats free gen­er­a­tors at equal com­pute on two model fam­i­lies.

Founders with deep expe­ri­ence in a thin-supply niche (Part 1) have a rea­son­able alter­na­tive: sell­ing ver­i­fied tasks in that niche to labs.

The kill-test clock

Figure 24 puts every area’s go or kill gates on one clock, in weeks from the week test­ing starts, beside an exam­ple set of dated events that force buyers to act.

Three read­ings matter. Of the seven gates in month one, three need no out­side party (integrity’s recall test, the luck net-effi­ciency test and the SQL trap card) and a fourth needs no buyer. The ear­li­est gate that needs a sig­na­ture is the cer­ti­fied migra­tion’s week-2 gate, which is why a team test­ing that line needs a warm part­ner. And in the exam­ple cal­en­dar the forc­ing events start at once. OpenAI’s Evals prod­uct goes read-only on 31 Octo­ber and shuts on 30 Novem­ber 2026. The dead­lines then clus­ter between Jan­u­ary and April 2027: the two OpenAI shut­down cohorts, the FCA’s oper­a­tional-inci­dent and third-party report­ing require­ments from 18 March 2027[204] and the Ops­ge­nie shut-off on 5 April. Fur­ther out, the EU AI Act’s high-risk obli­ga­tions apply from 2 Decem­ber 2027[205], and Arti­cle 15 requires that accu­racy levels and met­rics “shall be declared”[206]. Each date leaves some buyer need­ing evi­dence it cannot pro­duce itself.

The com­pany-level test. After the next shut­down cohort, at least one cer­ti­fi­ca­tion sub­scrip­tion has to renew into a second forced migra­tion. If none does, buyers are treat­ing the neu­tral com­par­i­son as a fea­ture of some­thing else, and the right move is to stop or to license the core to the kind of firm that buys mea­sure­ment, an infer­ence cloud or an observ­abil­ity incum­bent (when Data­dog bought Eppo, its vice-pres­i­dent of prod­uct said exper­i­men­ta­tion enables “teams to com­pare mul­ti­ple models side-by-side”[192]).

What would change the order

• Toward integrity first: a named lab writes a lab-appointed, third-party cer­tifi­cate into a vendor con­tract, turn­ing in-house re-cer­ti­fi­ca­tion into a pur­chase with a payer.
• Toward pri­vate knowl­edge sooner: the first audits show a fron­tier model hold­ing the cus­tomer’s own fact table as tools still erring on 10% or more of sin­gle­ton, reverse and multi-hop items above the A/A floor, and trained weights halv­ing that.
• Against the cer­ti­fied migra­tion: a lab’s deploy­ment arm cer­ti­fies its own cheap tier on cus­tomers’ tails with inter­vals, or an infer­ence cloud pub­lishes release cards with noise bands and tail slices.
• Toward a bigger com­pany: cer­tifi­cates become a pro­cure­ment norm, because insur­ers extend the cover they now write against agent cer­tifi­cates to envi­ron­ment and eval cer­tifi­cates, a model-risk val­ida­tor accepts a vendor-sup­plied paired eval­u­a­tion, or a reg­u­la­tor accepts agent-pro­duced evi­dence; or a ver­dict becomes some­thing the mea­sured party needs to sell, as a public rank­ing already is. Every larger esti­mate in this arti­cle assumes one of these.

Ten rules to print

Hold the busi­ness plans to the stan­dard this arti­cle holds scores to. That evals and envi­ron­ments are a large and grow­ing market is true, and it decides noth­ing for a founder. Pressed against its strong­est objec­tion, every one of the eight plans turned into a ver­dict busi­ness, and the buyer worth build­ing for is spe­cific: one who must re-val­i­date by a date, has some­one down­stream who will rely on the answer, and cannot get a neu­tral, rerunnable ver­dict from anyone with a stake in it. The busi­ness worth build­ing serves that buyer and holds its own num­bers to the stan­dard it sells. In RL envi­ron­ments, the durable prod­uct is a number that some­one with­out a stake in it can rerun.

Notes

1. Basis: the largest buyers are described secondhand as spending “$10M+ for single environments, with cumulative spend on the order of hundreds of millions per year”[13], read here as $0.2–1B for each of the two largest; four to six other frontier labs, neolabs and open-model developers at an assumed $25–150M each bring the total to about $0.5–3B (derived; the per-lab figures for the others are an assumption). Wing VC’s “tens of millions” for one lab’s 2025 spending with coding and computer-use vendors, growing 3–5× into 2026, sits at the floor. The band is 6–35% of the $8.5B that data vendors bill. ↩
2. 500,000 tokens per task (a one-year-ahead projection made in August 2025) × $15 per million output tokens (a retail price) × 64 rollouts = $480 a run; five reuses, an assumption, make $2,400. ↩
3. Inputs: engineers at $150 an hour loaded; experts at $110–250 an hour; calibration yield 0.65 (0.70 for law, 0.55 for security); review pass rate 0.90; 15% overhead; 56 rollouts per draft; one stateful environment at $111,682. Assumptions, checked only against the public cross-check in the text. ↩
4. ($948,027 cost of goods − $150,000 licence) ÷ 300 tasks ≈ $2,660 (derived). ↩
5. 300 tasks × 2 reviewers × 0.75 hours × $200 an hour. The exact zero-failure bound, 1 − 0.05^1/n, gives 5.8% at n = 50 and 0.99% at n = 300. ↩
6. Lean, middle, heavy: 5, 5 and 10 rollouts per task at $2, $3 and $4 (one agent leaderboard ran 21,730 rollouts for about $40,000[101]); deep review of a 10% sample at $115–$220 an hour; 20–40 hours of sign-off; sales and failed pilots excluded. ↩
7. Assumed: the screening model is right on 90% of sound items and fails 90% of broken golds; screening at $3.79 a query[109]; expert audit at four items an hour and $150 an hour[110]; making the pack costs about $156,000 at 14% broken golds and $102,000 at 2%. 14% is the share of one large synthetic text-to-SQL corpus that its own expert review did not judge fully correct[111]. ↩
8. Intercom Fin Apex 1.0 and Bridgewater on Tinker (both above); Checkr on Predibase[126]; FireRouter with Opus[127]; NVIDIA’s NeMo Switchyard router[128]; Genspark on Fireworks reinforcement fine-tuning[129]; and the hindsight router cited below. FireRouter’s ±19-point interval is on cost, not accuracy. ↩
9. With 10% discordance, 80% power and a Bonferroni correction across five slices, each slice needs n = (2.576 + 0.842)² × 0.10 / δ²: 467 items at δ = 5 points and 1,298 at δ = 3 (derived). Items cost $11–$40 each at $110–$200 an hour and 6–12 minutes per item with a second rater (assumption; one expert marketplace paid an average of $110 an hour in October 2026[138]). ↩
10. Wages and benefits alone come to $194,257 a person-year: the median US software developer wage of $135,980 (May 2025[154]) divided by the 70.0% share of wages in employer compensation costs (June 2026[155]) (derived). This article uses $300,000 a person-year throughout, for engineers paid above the median plus overhead (assumption). Compute: four agents run three times on each of 150 to 180 incidents at the midpoint of $6.50–$8.45 an investigation. ↩
11. How the cells were built, bottom-up: integrity, $240M serviceable less in-house lab work, committed evaluation budgets and compliance work that large assurance brands take, giving $34M; one firm, ten lab or vendor accounts at $0.3M plus 30 enterprise audits at $50,000. Cost frontier, a count of 80–180 workloads large enough to carry an endpoint, times fees; one firm, 20–40 certified workloads plus 5–15 endpoint workloads (an illustration). Operations, 100–300 multi-vendor operators at $75,000–$200,000 a year plus 10–20 AI-ops vendors buying independent evaluations, with almost no lab demand; one firm, 15–30 accounts. Decisions under luck, six buyer segments (labs, arenas and marketplaces, banks, broker-dealers, insurers, EU investment firms) times assumed shares and contract values, central case about $45M; one firm, a 10% share of that. Failure map, not sized; a three-person version costs about $104,000–$156,000 a month in cash, so it breaks even at roughly 210–310 team licences at $6,000 a year. Private knowledge, not re-sized after its objections. ↩