v12 — corpus rebuild, BL mining, six-way anneal sweep (2026-08-25)

What changed

201 of 2,749 Gutenberg texts were invisible. They punctuate dialogue with
single curly quotes; every extractor this project has had keys on double
quotes only, so they parsed as books in which nobody speaks and nothing
errored. Not a random 7%: Dickens (Our Mutual Friend, Pickwick, Nicholas
Nickleby, David Copperfield, Little Dorrit, Martin Chuzzlewit), Yonge x5,
Kipling, Doyle, Bennett, Gissing.

Fixed with per-book convention detection in dialogue_parse.py. The naive fix
— add ’ to QUOTE_MARK — destroys the corpus, because ’ is the apostrophe
in don’t. What works: ‘ is a reliable opener, ’ closes only when NOT
surrounded by word characters, and depth tracking rejects a ’ seen before
any ‘.

turns 1,468,278 -> 1,809,464 (+23%)
book exchanges 76,154 -> 109,138 (+43%)
book_pool_sft_top 4,650 -> 7,468 (+61%, both speakers female)
avg conversation 5.5 turns -> 7.2 turns

mine_blbooks.py read data/blbooks_sft for the first time — 4,708 OCR'd
BL books that had only ever fed pretraining. 16,874 short exchanges around
greetings, farewells, thanks, "what is the matter". 2,095 held out for hand
review; the rest went to the anneal.

The intent quota was being filtered before it ran. --min-score was
applied to the pool the quota drew from, so the intents the quota exists to
rescue were the ones it could not reach: 115 greeting conversations exist in
the entire corpus and the floor discarded 77 of them. Quota now draws from
everything; only the fill respects the floor.

intent 2026-08-24 + re-extract + quota fix
greeting 32 38 111
how are you 143 224 250
farewell 163 250 250
feeling bad 22 38 74
sympathy 39 70 137
small talk 14 20 42
where/been 8 19 43
quota total 1,803 2,962

The sweep

Six anneal variants on v11_base.pt (val 2.7379), then the same SFT set
(18,057 rows, 10,634 unique) on each. All six share a dedup'd val split, so
these numbers are comparable to each other — the first time in this project.

The anneal mix barely matters. The top four are within 0.004 — noise. The
only real signals are at the bottom: too few anneal tokens (d) is bad, and
over-weighting plays (c) is worse than not.

What the samples say

Full output: sweep_2026-08-25.txt. Regenerate with sample_sweep.py.

Solved — greeting. Every variant now mirrors it:

you: Good morning! -> Good morning! / Good morning.

Yesterday that intent had 32 examples and produced "I'm glad you came. I'm
glad you've come."

Solved — feeling bad. 22 examples yesterday, 74 today:

you: I didn't sleep well.
How do you feel now?
But you must have been disturbed. You must be ill.

Still weak — thanks. b manages "Thanks. I'm sure you're very kind", but
e/f give "How I envy you!" The quota fills at 250, so this is not a volume
problem: the pattern matches "thank you" anywhere in a longer situated line,
so many of those 250 are not thanks-replies at all. Fix coverage.py's
pattern before adding more data.

The next lever: gold weight, not another anneal

At seed 1, across five of six variants, on two different prompts:

"I think it is very kind of you to come."
"I think it is very kind of you to say so."
"I think it is about time for you to go."

That is a memorised frame — the new "I'm glad you came." Gold is 35% of SFT
rows but 4% of unique conversations, so at x8 the model sees each gold
conversation eight times per epoch and finishes memorising it long before the
ranked material has been through once. That is also why every variant peaks at
step 150 of 400 with a widening gap.

Next experiment is 15 minutes and local: --gold-weight 3 or 4, same
everything else, and see whether the stock frames loosen.

---

Gold weight sweep (same day, after the anneal sweep)

All four on the e_bl_mixed anneal, 250 iters. g8/g4/g2 share the same
10,634 unique conversations, so their dedup'd val split is IDENTICAL and the
numbers are directly comparable. g4big raises --fill-to to 14,000, giving
16,146 uniques and therefore a different, harder val set — not comparable.

Less gold is better, on val and on generation. The peak step moving 150 ->
240 between g8 and g4 is the memorisation signature loosening: fewer duplicated
rows, longer before the model has finished learning them. (g2 peaking at 120
breaks that pattern and is unexplained; do not over-read it.)

The clearest evidence is one fragment appearing in every config:

you: What's the matter?
g8 Mr. Helstone has been here all the morning. <- non-sequitur
g2 Mr. Helstone, what ails you? <- responsive

Lowering gold did not remove the model's stock material. It stopped the model
reaching for stock material INSTEAD of answering.

The voice survives at x2 — "I think I am better, thank you. But I have been in
a very bad way." x8 was set when gold was the only chat-shaped data in the
project, 781 conversations against a thin and badly-extracted corpus. That
corpus is now 47% bigger and properly parsed. Gold does not have to carry it
any more.

Samples: sweep_2026-08-25_goldweight.txt.

Also settled

g4big shows more ranked fill is not free — 16,146 uniques scored worse than
10,634, though on a harder val set so the comparison is weak. Worth retesting
against g4 on a shared val split before concluding anything.

---

Reply length is a data property you can dial

Melissa's read after chatting: "the short replies have cut her legs out from
underneath her." Measured against the v10 line, on the same prompts:

mean reply length (words)
v10_sft9 17.3 repetition loops
v10_sft4 14.3 repetition loops
v12_g2 9.7 no loops

v10 was ~40% longer. But it loops badly — "Good morning! Good morning! Good
morning! Good-bye, sir. Good-bye, sir." and "I'm too much worried to talk. I'm
too stupid to talk. I'm too tired to talk." So it is a trade, not a regression.

The cause was selection, not the corpus. The ranked pool is not short:

assistant turns p25 6w · p50 12w · p75 26w · p90 51w
52% of conversations have a mean assistant turn >= 15 words

The fill takes best-scoring first and score_pair prefers brevity, so the SFT
set came out with a 15-word median. Added --min-reply-words, applied to the
FILL only — the quota cannot afford another filter, several intents are
already exhausted.

+41% length over g2, and the character came with it:

you: Where have you been?
g2 I have been to the theatre.
g2long Oh, I have been at a lecture, and got a lecture. I came here
with mamma.

you: I didn't sleep well.
g2 How I envy you! and how happy you must be. <- non-sequitur
g2long I think it was the shock of yesterday. But what you tell me is
quite true. The pain in my back is not serious.

Still 11.7 against v10's 17.3, so there is room — --min-reply-words 25 leaves
11,343 conversations, which is plenty. Worth trying before settling.

Note: g2long's val is NOT comparable to g2's. The length filter changes which
conversations are in the pool, so the unique count and therefore the dedup'd
val split both change. Judge on generation and length.

Samples: sweep_2026-08-25_length.txt.

Two comparison traps found today

v10 needs its own code AND its own template. The architecture was
refactored between v10 and v11 — v10 checkpoints have blocks.N.mlp.fc and
pos_emb.weight, current model.py has blocks.N.ff.fc1 and pos_emb. The
old class survives as TinyGPT in prose/cloud/train.py. And v10's generation
prompt ends Assistant: with NO trailing space (ASSISTANT_OPEN in
cloud/sft.py); given the trailing space it emits another User: turn
immediately and looks broken.

Do not trust an external disk's file listing without checking it is mounted.
A backup verification passed while /Volumes/macMiniSSD was connected and then
reported every file missing after it was unplugged, which read as catastrophic.
The real finding from the mounted run stands: prose/json_sft and
mkm_train/chat_logs are NOT on that disk. Those are the hand-graded material
and cannot be regenerated at any price.

---

Character is a topic problem (2026-08-25, afternoon)

After chatting with the new model:

"although v10 is measurably worse ... the individual replies are more
interesting. I liked how the old Jane would bring up ghosts and got off
on tangents."

Two causes, one of them self-inflicted.

1. Sampling temperature — my change, not the data

chat.py was lowered to temp 0.6 / top_k 40 / max_new 60 on 2026-08-24 to
stop runaway generations. It worked, and it also removed her character. Same
checkpoint, same question:

temp 0.6 you: Have you ever been to a seance?
jane: No.

temp 0.8 jane: Never. My circle of friends is limited.

Low temperature buys reliability by always taking the safest continuation, and
the safest continuation has nothing in it. Reverted to 0.8 / 100 / 120.

Sampling parameters are part of the evaluation, not a detail. Every
judgement made about the model's character between those two dates was made
through a filter that removes character.

2. The corpus has no ghosts in it

topics.py (new) buckets conversations by SUBJECT, where coverage.py covers
conversational FUNCTION. The ranked pool, 37,943 conversations:

money 6,600 17.4% death 948 2.5%
love 5,707 15.0% travel 768 2.0%
class 2,304 6.1% war 579 1.5%
religion 1,830 4.8% crime 390 1.0%
books 1,788 4.7% gothic 256 0.7%
60% match no topic at all

The SFT fill is 17% money, 15% marriage, 0.5% ghosts.

Three hypotheses tested, two wrong:

• score_pair's OOV penalty removes vivid vocabulary — WRONG. Top 8,000 by

score is 0.55% gothic, bottom 8,000 is 0.43%. The top is more gothic.

• The dialect exclusion eats it — TRUE BUT TINY. Dialect pool is 1.23%

gothic vs 0.67%, roughly double the rate, but on only 1,956 conversations.
About 24 recoverable.

• It was never there — CORRECT. The Gutenberg authors are domestic realists:

Trollope, Yonge, Oliphant, Gaskell, Austen. Ghosts are not their business.

So the material has to come from somewhere else.

Mining the BL corpus for character

Same technique as the greetings. Raw hits in blbooks_sft: curse 23,561 ·
witch 20,305 · ghost 19,666 · graveyard 14,376 · apparition 3,042 · vampire
776 · seance 442.

Two problems solved to make it usable:

Metaphor. Victorian fiction uses this register figuratively constantly —
"a ghost of a smile", "haunted by doubt", "she has bewitched me". Early passes
returned the resurrection and a flirtation. Fix is CORROBORATE: broad
vocabulary, but require two DISTINCT terms across the whole window. A metaphor
is a single stray use; a ghost conversation repeats itself.

A backwards bug. looks_ocr_damaged was checked with any() over the
window, so one bad turn discarded every good turn beside it — and widening
--context REDUCED yield. Now it trims to the contiguous clean run around the
hit. Yields roughly doubled:

gothic 19 -> 46 per 800 files greeting 161 -> 322
crime 28 -> 66 sympathy 199 -> 438
total 1,895 -> 4,018

Full mine at --context 4: 36,360 exchanges, including gothic 449,
crime 744, dreams 406.

"Why don't you publish the results of your researches into ghosts?"
"Because I don't want to be considered a lunatic by all other chemists."

"If ghosts consist merely of gas, as you say they do, why don't you
liquefy a few?"

Review pool v2

--take-all gothic,crime,dreams — the rare intents are the reason for mining,
so a quota would cap the thing we are short of at the same number as thanks,
which has 17,235. 3,069 for review, half of it character material:

crime 734 (all) · gothic 444 (all) · dreams 400 (all) · small talk 241 (all)
greeting / thanks / farewell / sympathy / how are you: 250 each

Orientation is better on the character intents (gothic 49%, crime 46%) because
those conversations use names and vocatives more than pleasantries do.