Jane from scratch vs Qwen2.5-0.5B on the same corpus

2026-10-05. Settled. Read this before proposing to re-run it.

The question

Not "how do I make Jane good" — seven data-mixture levers came back flat and
that one is answered. This asks what from-scratch actually costs you, by
putting a 245M model trained on nothing but Victorian prose against a modern
0.5B base fine-tuned on the identical file, graded by the same blind grader.

What was held constant

The one variable left is the base model. Second axis: sft_r4 (fill ordered by
score_pair) against sft_randfill45 (random above score 45).

Result

40 single replies, blind, shuffled across all four models.

qwen_r4           7.60   sd 1.84
randfill45        7.40   sd 2.32
r4                7.20   sd 2.30
qwen_randfill45   6.90   sd 2.73

Paired on opener, nothing is significant — largest |t| is 1.11.

spread of MODEL means    0.70   (sd 0.30)
spread of OPENER means   5.00   (sd 1.68)
residual sd              1.53

The opener matters 5.6x more than the model.

What this licenses saying, and what it does not

Does: a 245M model trained from scratch on 7.37B tokens of Victorian prose
is not detectably different from a 0.5B pretrained on trillions of modern
tokens — same data, same supervised budget, first replies.

Does not: that they are equivalent. With n=10 and a paired sd of 2.54 the
smallest detectable difference is 2.33 points on 0-10. This could only ever
have found an enormous effect. It is a bounded null, not a measurement of zero.

Also does not: anything about multi-turn. Single replies were chosen
because ouroboros.py cannot put a Qwen in Jane's seat, and because turn one is
where responsiveness shows. Informal chat suggested Qwen tracks facts across
turns better (it picked up "my father is a carpenter" and answered it) while
confabulating just as hard. Untested. That is the live hypothesis if this
is ever picked up again: pretraining buys coherence, the corpus decides
groundedness.

The opener finding, which is the durable one

Third independent replication:

Different models, different data, different task format, same answer. Any
future comparison needs more openers, not more repeats per opener.

Why not just grade more

To resolve a half-point difference at this variance takes roughly 200 openers
x 4 models = 800 graded items. That is the grind that makes this project stop
being fun, spent on a distinction that probably does not matter. The comparison
is finished, not pending.

Bugs fixed on the way here, so they are not rediscovered

All of these produced garbage output before anything was measurable:

1. mask_prompt: false — trained on user turns as targets, so it generated

both halves of the conversation.

1. mlx-lm masks all but the last message, so one 7.6-turn conversation

supervised one turn. Fixed by expanding to one example per assistant turn.

1. --system-prompt "" at chat time against a template-injected

"You are a helpful assistant." at train time.

1. Raising LR to 1e-4 — unjustified. mlx-lm's LoRA scale is 20.0, so 1e-5 is

not the timid setting it looks like.

1. 5.2% of expanded examples had zero supervised tokens — truncation at

--max-seq-length cut at or before the mask offset, the loss divided by
zero, and the adapter was NaN within ten steps. This is what produced
!!!!!!!!. prep_mlx.py now drops them.

1. Streaming the reply then redrawing the trimmed version with \r printed it

twice; \r returns to the start of the current line, not the output.

And one that is not fixed and should not be chased: mlx-lm's
--fine-tune-type full does not train the embedding, so Qwen cannot learn its
stop token. See stop-token.md. It is tooling, not a property of the model,
and must not be reported as a difference between them.