Tiny prose dpo round1

First DPO round, 2026-08-13: dpo.py from model_v10_sft8/best.pt, 4,136 pairs
(1,671 live, 40%) from 1,417 graded turns, 877 iters / 2 epochs, ~25 min local.
Result: no detectable improvement, and the checkpoint was deleted.

Three independent measures agreed, which is why this is settled and not worth
re-running on a hunch:

• val ranking accuracy 60.1% → 61.2% against a ±2.0% s.e.
• edit-pair accuracy 43.8% → 45.4%, still under 50% — on held-out turns it

still prefers the text she edited away to her rewrite

• blind A/B on the 5 v6_v8_probe.jsonl prompts: dpo1 3, sft8 2 (a coin flip).

Only 4/30 pairs byte-identical, so this was a fair test of a genuinely
different model, not two near-identical checkpoints.

Why it failed, and why more aggressive settings would be worse: the
training dpo term fell 0.689 → ~0.44 while held-out accuracy stayed flat.
It memorised preferences on the turns it saw. Raising --lr or lowering
--alpha accelerates that. What it needs is more distinct graded turns, not
more pairs squeezed from the ones already graded.

Also measured: dpo1 replies averaged 18.9 words to sft8's 22.0 (max 53 → 38).
The NLL anchor concentrates mass on safe phrasing, which showed up as
two-part replies that contradict themselves ("I'm very sad... It is not
possible to be sad"). Watch reply length on any future preference run.

Fixed along the way (keep): grade_conversations.py stamps the text she
edited away at 0.5, which USABLE_MAX = 0.5 put on the keep side — so all
738 hand edits were paired with their originals instead of against them.
ref_logprobs.py now emits is_orig and both scripts split via a shared
is_keeper(). That recovered 782 rewrite-vs-original pairs at the highest
live rate in the set (53%). The fix is right regardless of this run's outcome.

Related: tiny-prose-corpus-retrieval (the intervention that did work,
4–0 blind), tiny-prose-side-project, feedback-billable-commands.