compare/

Jane from scratch, against the same corpus fine-tuned onto a modern base.

The question is no longer "how do I make the from-scratch model good" — seven
data-mixture levers came back flat and that one is answered. The question here
is what from-scratch actually costs you, measured on the same corpus, the
same openers, and the same blind grader.

Start at Qwen2.5-0.5B — near enough in size (500M vs 245M) that the result
reads as a claim about pretraining rather than about parameter count. The
larger ones come after, as a separate row.

---

Result, 2026-10-05: notes/results.md. All four cells came out even — the
largest paired difference was t = 1.11 — and the opener mattered 5.6x more than
the model. The comparison is finished; read that before proposing a re-run.

Where things go

One rule, and it is structural rather than a matter of discipline:

git holds what you wrote. runs/ and models/ hold what the scripts produced, and they are ignored.

compare/
  src/              shared code — bench driver, blind grader, jev harness
  notes/            findings
  <model>/
    conf.json       what the run was configured with
    runs/           every generated artifact        (ignored)
    models/         adapters, checkpoints           (ignored)

.gitignore carries compare//runs/ and compare//models/, written before
anything had a chance to land anywhere else. That is the whole fix for how v11
ended up with 104G of checkpoints in the tree and 252G in .git: output had
nowhere of its own to go, so .gitignore chased individual paths and always
lost. Here there is nowhere else to put it.

Backup is therefore one path per model, not six.

---

What is held constant

The whole experiment is one controlled variable: the base model. So
everything else is pinned, and the training data above all.

both models train on the IDENTICAL file — prose/json_sft/built/sft_r4.jsonl

jev does not get to select or re-rank the input for this comparison. A
better-chosen corpus for one side would mean the difference attributes to the
data and the base at once, and neither could be read off the result. jev grades
OUTPUT here; it is an instrument, not a treatment.

(Using it to choose data is still a good idea — just as its own experiment,
run on both sides or on neither.)

This has a consequence worth stating plainly: the Jane side of the comparison
must be r4, not r3b. sft_r3.jsonl was overwritten on 2026-08-29 and no
longer matches what r3/r3b/r3long trained on, so there is no way to give Qwen
r3b's data. r4's dataset is intact and timestamped four minutes before its
checkpoint, so r4 is the one Jane that can be matched exactly.

r4 was deleted on 2026-10-03 during the cleanup, on the grounds that it was
regenerable in ~10 minutes from sft_r4.jsonl. Regenerating it is now step one,
which also tests that claim.

---

Grading

Three opinions on the same conversations:

1. Blind human — v11/scripts/score_bench.py, shuffled across models,

0-10. Already calibrated; 299 grades exist.

1. jev (typesafe-sdk) — calibrated floats, parallel per question.

Hosted, so billable, so Melissa fires those calls.

1. The existing defect metrics — chat_audit.py, kept only as a floor.

jev gets validated before it gets trusted. Every sensible-looking metric
tried so far has run backwards against her ear — self-contained and
responsive both did, which is why chat_audit reports AUC against the
keep/cut verdict rather than a raw score. So the first jev run is not over the
corpus; it is over conversations that already carry a human grade:

v11/_ouroboros/scores.jsonl 299 blind 0-10 grades
prose/json_sft/ready/graded_batch400.jsonl 233 hand-graded conversations
chat_audit.KEEP the 8-keep / 10-cut verdict

If jev's float tracks those, it can grade thousands and go to work on the input
side, where score_pair's 5-points-per-word cliff is the weakest link in the
pipeline. If it does not track, that is worth knowing for a few dollars rather
than three weeks.

---

Talking to them

Nothing new was written for this — v11/chat.py and mlx_lm.chat both already
do it.

# Jane
venv/bin/python v11/chat.py --checkpoint compare/jane/models/r4/ckpt_best.pt
venv/bin/python v11/chat.py --checkpoint compare/jane/models/randfill45/ckpt_best.pt

# Qwen
HF_HOME=~/Models/hf-cache ~/.local/bin/mlx_lm.chat \
  --model Qwen/Qwen2.5-0.5B --adapter-path compare/qwen/models/r4 \
  --temp 0.8 --system-prompt ""
HF_HOME=~/Models/hf-cache ~/.local/bin/mlx_lm.chat \
  --model Qwen/Qwen2.5-0.5B --adapter-path compare/qwen/models/randfill45 \
  --temp 0.8 --system-prompt ""

chat.py defaults to temp 0.8, top-k 100 and repetition penalty 1.15, which
ride inside the checkpoint. --temp 0.8 on the Qwen side matches it.

--system-prompt "" matters. Qwen2.5's ChatML template injects "You are a
helpful assistant." whenever no system message is supplied, at training time as
well as here. Jane has no system prompt at all, so leaving Qwen's default in
place would be comparing a model that was told to be an assistant against one
that was not. Empty on both sides, or the difference includes the instruction.