Tiny prose decoding config
For Little Jane, decoding settings have twice been mistaken for model quality.
Both fixes landed 2026-08-26 and neither required retraining the base:
- Repetition penalty 1.15 (CTRL-style, ported from v10's
generate_candidates.py). Nothing in the v11 path had one. Adding it moved
mean reply length 9.3 → 12.2 words on unchanged weights and ended the
run-on loops.
- A learned stop.
</s>(id 2, already in the tokenizer, previously
unused) now closes every assistant turn in sft.py:tokenize() and is
supervised. Learned 24/24 in 250 steps because it is 3.1% of all supervised
tokens. Before it, replies were cut by string match or truncated mid-word
by max_new.
Why: a repetition penalty cannot be trained into weights — it is applied
to logits at sampling time. But "remember it every time" is a packaging
problem, not a law. sft.py now writes a sampling block into the
checkpoint and chat.py reads it. Same lesson as
mlx-lm-gguf-ollama-toolchain: weights are half a model, the serving config
is the other half.
How to apply: when she reads flat, clipped, or looping, check decoding
before blaming data — including the harness itself (sample_sweep was also
clipping its own printed output at 140 chars, which looked identical to a
truncated generation). Checkpoints before 2026-08-26 carry no sampling
block; pass --repetition-penalty 1.15 by hand. See
tiny-prose-side-project and prose/notes/sft-loop.md.