The Model Was Fine.

Forty hours. One rented GPU. About $170 of compute (DigitalOcean GPU Droplet pricing). Twenty-four attempts to make a language model better at the work my company actually sells. At hour thirty-eight I had one that worked. At hour forty I took it back.

I want to describe what those forty hours looked like, because I do not think most people have a picture of it. When you read that a company trained its own model, you probably imagine a laboratory. It is not that. It is one machine you rent by the hour, a list of recipes, a set of test questions you wrote yourself, and a lot of waiting. You start a run. You go do something else. You come back and look at a number.

Everything rests on that number. This is the piece where I tell you that for forty hours, my number was wrong.

Part one of this series named three levers, training, harness and evaluation, and argued that the cheapest lever on any given day is almost never the training lever (Three Levers, One Loop). That was the framework. This is what happened when we ran it long enough to catch it lying.

Under the ruler that ran the program, Round 11 landed at 0.65 on the production path and cleared a four-point promotion margin over the base model at 0.61. It was the only round in twenty-four that cleared anything.

Then we audited the ruler. It had four defects: ordinals counted as invented facts, five workbook cases forced onto a routing slot the candidate never sees, a pass-or-fail resolution too coarse to detect a real signal, and equivalent numeric forms counted as disagreement. Corrected for those four, the candidates land inside a two-case band, the base model sits inside that band, and Round 22, which the old ruler had recorded as sitting behind production, becomes the leader. The ranking reverses on a change to the instrument, not a change to the models.

Strictly, no comparison in the program was statistically significant, including the promotion. The Wilson 95 percent half-width on a 70 percent baseline at 48 cases is roughly 13 points (AI Evals · Statistical methods). The margin we were trying to steer with was 4. We were reading noise and calling it progress.

Part one said the loop should be built to earn one thing: the confidence to know when your own ruler is lying to you. The loop earned it. It cost forty hours and one deleted promotion.

The whole program on one line makes the shape of it legible in a way the raw log does not.

The training lever ran continuously, twenty-four tick marks across the strip, one every ninety minutes on average. Four harness changes went in over the same window, each a few hundred lines of deterministic code. Two evaluation events sit at the ends. The golden set grew from 24 cases to 48 in the first six hours, because the smaller one had stopped telling recipes apart. The instrument audit ran in the last two.

The production number moved from 0.52 to 0.65 across the program. Four points of that came from the harness fixes, and those held when the audit came. At most six points came from training, on weights that had already shown a three-point range from nothing but a different data shuffle. The audit reversed the training attribution. The rest of the movement is the baseline shifting when the golden set doubled.

That is what a training program actually looks like when you instrument every layer. Not twenty-four fine-tunes running in isolation, but twenty-four fine-tunes running alongside a harness under active engineering and a measuring instrument being sharpened underneath both. The training lever is one of three, and in this program it was not the one that moved the number.

The instrument had four problems. Any one of them was enough to explain the ratio of promotions to noise. Together they explain why twenty-four rounds produced exactly one promotion, and why that promotion did not survive being looked at.

Two of the four fixes were free. They are data we already had, counted differently. Scoring the roughly 300 assertions inside the 48 cases instead of 48 binary outcomes sharpens resolution about fivefold, and pairing runs on the same cases rather than as independent totals removes the variance the two runs share (Wolfe · Applying statistics to LLM evaluations; AI Evals).

The third fix is the one that made the audit believable at all: a held-out set drawn from real shipped work, assembled before any of the fixes were applied. The instrument we ran the program with had been optimized against, quietly, for every round it scored. The new one had not.

Four of the twenty-four rounds are worth returning to. Round 11 was the promoted one, a light LoRA on the recipe that had been working. Rounds 22, 23 and 24 were the three reinforcement-learning rounds, trained against a reward a program can verify rather than a preference model's opinion.

Round 22 was the lightest touch in the program. One sweep against a length-verified reward, with a KL divergence around 0.001, which means the policy barely moved off the base model at all. Round 23 was the same idea at full dose. Round 24 doubled the reward's length ceiling to answer a problem Round 22 had created, and produced 527-word outputs against a 264-word target.

Two things are true about that scoreboard at the same time. On the ruler that ran the program, Round 11 leads and looks like a promotion. On the corrected instrument, Round 22 leads by two points, Round 11 sits second, and every one of them is inside the base model's confidence band. Both readings are real. The only thing that changed between them is the ruler.

The Round 22 result is the one mechanically interesting thing in twenty-four rounds. The lightest possible reinforcement-learning round, from the smallest policy change, produced the best bare-model profile of the program, and every heavier version of the same idea did worse. That is a specific, testable claim about where to spend the next training lever. It is also the only claim from the training work that a corrected instrument still supports.

The other half of the forty hours had nothing to do with the model.

For two full days, financial workbooks failed every case in every round under every recipe we tried. I wrote it down as a capability wall. The model cannot do workbooks. That was wrong.

Nobody had ever written down what a finished workbook is. The requirement existed in exactly one place, inside the grader, where the model could not read it. We wrote the definition down, wired it into the same check-and-repair loop that had already fixed the memo class, and workbooks went from zero to sixty percent in a single deploy. Same weights. Same model. The only thing that changed is that the standard now existed in writing.

Then we pointed the production model at real deliverables from the business. Investment theses, business cases, structuring memos, term sheets, diligence responses. Actual work, the kind we sell.

It scored zero on the first seven.

The cause was the same every time: length. Output ran 165 to 367 words against requirements of 398 to 700. This was not a truncation limit. The generation budget allowed roughly ten times that. Contract coverage across real deliverable types was nine of eighty-six, about ten percent. The work the business actually sells had no definition of finished, so nothing detected short output on those classes, and nothing repaired it.

If you have ever handed work back to a person and said this is not what I asked for, you already know this problem. It is rarely that they could not do it. It is almost always that nobody wrote down what finished means.

What we were short of was specification, not capability. That is the load-bearing sentence of the whole program. Every hour I spent on the training lever before that sentence was true was an hour spent chasing a number that could not tell me what was wrong.

On Sunday morning, once the pattern was clear, we generalized it. Six enterprise deliverable classes each got a written contract: a minimum word floor, a minimum number of figures, and a required section list. Plus one rule that does more work than the other six combined. When a request names its own structure, that named structure becomes the contract.

The first held-out measurement of that six-contract deploy is running as this publishes. Part three reports the result, whichever way it lands.

The strategic asset in this program is not ownership of OLMo. It is the capability that OLMo's unusually complete open foundation let us build. That distinction is worth drawing out properly, because the two get collapsed constantly and they are not the same thing.

That last line is the case study. Not the checkpoint, not the benchmark, not any single round in the twenty-four. Open infrastructure moved this company from operating a model to developing one, and those are different businesses with different capabilities.

A rented endpoint gives you inference. It does not give you a development pathway. Every hour described in this essay went into things a rented endpoint does not expose: what the grader is actually counting, how a candidate gets routed, what a finished deliverable is, whether the promotion margin is larger than the noise. You cannot audit an instrument you are not allowed to see.

It is also why the gains hold. The harness fixes, the contracts and the corrected instrument all attach to the implementation layer, not to the foundation. When a better open foundation arrives, and it will, every one of them moves with us.

The easy read of this program is that I spent $170 and forty hours and produced no working fine-tune. That read is wrong on two counts.

First, we produced a platform. The production number moved from 0.52 to 0.65 during the program, driven mostly by four deterministic harness changes that hold across every future model version and every future training round. That is not a fine-tune. It is a permanent capability, and it compounds. Every model we train from here runs through it.

Second, we produced a corrected measuring instrument, a characterized noise floor, a written specification layer, and one training result worth returning to. Those four things outlast any checkpoint. A promoted checkpoint gets discarded at the next base-model refresh. The instrument, the noise floor, the contracts and the Round 22 finding do not.

Rank the levers by cost per point that survived audit and the training lever finishes last. The harness lever cost engineering days and moved the number four points that held. The evaluation lever cost two hours and produced the audit that reversed our own promotion. The specification lever cost one written contract and moved a class of failures from zero to sixty percent in a single deploy. The training lever cost the most calendar time and the most money, and it produced nothing that survived being measured properly.

Twenty-four training rounds in forty hours is not a story about a fine-tune. It is a story about a loop, run for real, run long enough to falsify one of its own decisions. The loop diagnosed each failure at the layer responsible. The loop was willing to unpromote itself when the instrument turned out to be wrong. It produced a platform that got measurably better on real work, an instrument that can see whether the next round actually moved anything, and a shorter list of things worth trying next.

That is what an open foundation is for. A rented endpoint gives you inference and nothing to inspect. An open foundation, operated with your own data, pipeline, infrastructure, harness and evaluation, gives you an engineering surface that gets stronger every quarter. It also gives you the evidence, and the discipline, to conclude that your best trained model is indistinguishable from the one you downloaded, and to act on that instead of shipping the promotion anyway.

Forty hours. One GPU. Zero surviving promotions. One better platform, one sharper instrument, one written specification layer, and one training result worth going back to. That is what the loop is for. The weights are one output of it.

And the reason to run it at all is the one I started with. Somewhere in this company there is a person who spent part of today on something a machine should have done. Getting that hour back to them is the whole job. The instrument work is what tells us whether we actually did it.