Corpus design: pretrain is a language, SFT is a voice

The single most important rule in this project, and the one the fetch rounds
took five tries to arrive at.

The rule

Pretraining teaches Little Jane English — specifically, 18th/19th-century
English narrative prose. SFT teaches her who she is.

These two jobs want opposite things, and every corpus decision falls out of
which job the text is for.

Why author sex stopped mattering for pretraining

Rounds 1–2 fetched a corpus that was heavily Victorian women, and it did both
jobs at once — it was the pretraining data and the source of voice. That
conflation is what capped the corpus at 198 authors.

It's the wrong model of what pretraining does. A 243M model that reads
male-authored Victorian prose does not come out sounding like a man. It comes
out able to produce Victorian English sentences, which is the only thing
pretraining at this scale reliably installs. Voice is washed out by volume —
that is precisely why it has to be installed later, in SFT, where a few
thousand curated examples are enough to set it.

So filtering pretraining data by author sex costs roughly half the available
tokens to buy a property that pretraining cannot hold. It is a pure loss.

The corpus is currently 50x under the conservative Chinchilla point for
243M params (95M tokens against ~4.9B). Anchor-phrase collapse, register
drift, and DPO round 1 producing no measurable gain are all consistent with a
starved model. Volume is the priority, and nothing may be traded for it.

Where the female voice actually lives

In SFT, and only in SFT. Every U/A pair Jane learns to produce should be a
woman speaking. That constraint is absolute and must never be relaxed to make
the SFT set bigger.

But note what the criterion actually is: speaker sex, not author sex.
A woman's line written by Dickens is a valid SFT example. A man's line written
by Gaskell is not. Author sex was only ever a proxy, and it is both too loose
and too tight — it lets male speakers in and shuts female speakers out.

Two consequences:

1. The SFT harvest pool is much larger than "books by women." Every female

speaker in the male canon is available. dialogue_parse.label_speakers
is what makes this decidable.

1. The BL shelf — overwhelmingly male-authored — is still a legitimate SFT

source, on its female speakers.

The fetch rounds, read back through this rule

Round 5's docstring already had the insight in it — "a pretraining corpus
wants the canon; a retrieval corpus wants everything the canon left out" —
it just hadn't been generalized to the whole pipeline.

Register band

Pretraining is bounded by register, not by date or sex: English narrative
prose, roughly 1700–1900. The band is wider than "Victorian" because 18th and
19th century English are far closer to each other than either is to the 20th,
and because there are few enough 18th c. novels that they land as a
single-digit percentage of tokens — flavor, not competition.

What stays out is anything the model would have to hold as a second
register: 20th century prose, and text that isn't Victorian English at all —
wrong language, wrong era, or a destroyed scan.

Victorian non-fiction stays in for pretraining. Travel writing, county
history and biography are still Victorian narrative prose, and at 50x
under-fed the volume is worth more than the purity. This was not the original
plan — the first draft of this note excluded "the non-fiction bulk of the BL
shelf" — and it changed on measurement: see below.

OOV is a garbage filter, not a genre filter

gate_oov.py measures how much of a book's vocabulary is missing from clean
Victorian prose. At the extreme tail that correlates with genre, and it was
briefly mistaken for a genre filter on that basis. Cross-tabulated against
dialogue density over 400 books, the two are close to independent:

               dialogue-rich   dialogue-poor
OOV <= 5%             265             107
OOV >  5%              19                9

29% of books passing the gate are dialogue-poor. Around the threshold it
sorts by scan quality, not by kind of book — at 5% it dropped
made_in_heaven_a_novel and kept a topographical dictionary of Wales.

Dialogue density can't stand in for genre either. probation_a_novel,
a_crown_of_gold_a_novel and rex_the_black_sheep_a_novel all have under
2 quote marks per 1,000 words, because the scan lost the quotation marks. A
dialogue gate at 8/1k costs 36% of tokens and takes those with it.

So: one filter, one job. The OOV gate removes non-English, non-Victorian
and destroyed text, at 8%. Dialogue density belongs in the SFT harvest, where
a book with no surviving quote marks genuinely has nothing to extract.

Two cleanliness bars

A corollary that saves an enormous amount of work.

Pretraining needs bulk readable prose. A stranded quote mark is noise across
670M tokens and averages out. The bar is: is this narrative prose, and is the
OCR readable?

SFT needs the dialogue extraction to be near-perfect, because each pair is
seen many times. All of dialogue_parse — paragraph restoration, quote
parity, attribution, narration detection — exists for this bar and this bar
only.

Applying the SFT bar to pretraining data is what made the BL shelf feel
impossible. It doesn't need it.

The tokenizer is trained on Gutenberg only

The decision that makes the whole BL plan safe.

BL's median book is 1.98% out-of-vocabulary against clean Victorian English;
Gutenberg's is 0.10%. Twenty times dirtier. What that noise costs depends
entirely on one choice:

• Tokenizer trained on Gutenberg + BL. SentencePiece sees com mittee,

tinued, selves, hav at high frequency and spends vocabulary slots on
garbage fragments. That degrades every token's representation, not just
the damaged ones. It is the one way a 2% noise rate does 100% damage.

• Tokenizer trained on clean Gutenberg only, then used to encode BL.

Broken words simply fragment into more pieces — com + mittee. The
vocabulary stays pristine. The cost is a few percent more tokens for the
same text, and the model averages the noise away, which is exactly what
pretraining at 600M tokens is good at.

Take the second. The trade being made is compression for vocabulary
cleanliness, and at a 2% noise rate cleanliness wins easily. 73M clean tokens
is plenty to fit a 16–32k vocab.

This is also why strip_accents.py has to be run on BL and not just
Gutenberg. Gutenberg's accents are already stripped, so a tokenizer trained
on it contains no æ — every one left in BL would be guaranteed OOV. Both
corpora must be normalised the same way for the split to work.