melissa kate
<< Little Jane

Tiny prose side project

Side project: from-scratch TinyGPT trained on Victorian women's writing, SFT'd for chat. The model is named Little Jane. Lives in prose/ (cloud/ = train+sft+generate scripts, scripts/ = data pipeline, models/ = checkpoints, data/ = corpora, json/ = chat/graded JSONLs). Philosophical companion doc co-written as we go: project_notes/CONVERSATION.md (séance/Frankenstein/time-travel framing — she's the medium, corpus = corpse, no living voices allowed in the channel).

Current state (2026-08-27)

Best checkpoint: v11/models/sft_cfg/g4voc2/ckpt_best.pt. The whole SFT
loop — build, run, listen — is written down in prose/notes/sft-loop.md,
including the reasoning for every current flag. Read that before changing a
setting; most of them were arrived at by measurement and one was reverted
twice.

Two days of work on 2026-08-26/27 found that most of what made v11 "sound
worse than v10" was scaffolding, not the model: a repetition penalty v10 had
and v11 lost, gold windowing dropped in the rebuild, no end-of-turn token, and
two places where the evaluation harness truncated her output in ways that read
as generation failure. See tiny-prose-decoding-config.

Live findings worth keeping:

conversation, not only of the data — a leading question gets 58 words, a
plain "how have you been" gets 30. Check the ratio before reaching for a
corpus dial. sample_sweep only tests turn one and understates this.

conversations in gold/continuous_conversations.jsonl, doing the work of
thousands — Mr. Helstone keeps surfacing because of it. BL grading fixes
the intent quotas; chatting fixes gold. Different levers.
prose/scripts/chat_to_review.py turns chat logs into a grading pool.

Current state (2026-08-19) — v11

The corpus was rebuilt from 95M to 1.02B tokens. This supersedes most
size/arch numbers below; the older sections are kept for the reasoning, not
the figures.

books) + pretrain_blbooks.npy 716.9M (7,031 books), uint16, EOS between
books. BL Books is BACK and is 70% of the corpus — the "dropped entirely
for OCR noise" call below is REVERSED. It was readmitted behind a 5-rule
vocab-gated repair pipeline (cleaning/clean_blbooks.py + gate_oov.py),
and the pretrain/SFT split is what makes it safe: pretrain only has to be
narrative prose with readable OCR, SFT needs near-perfect dialogue. See
tiny-prose-corpus-design.

tokens/param against Chinchilla's 20 — data starvation. Anchor-phrase
collapse, register drift, and the null DPO result are all consistent with
it. Problems previously blamed on curation were mostly starvation.

(weight tying, scaled residual init, SDPA), train.py (spread val windows,
warmup+cosine, decoupled weight decay, grad clip, bf16 autocast + TF32 +
--compile on CUDA), tokenizer.py, encode.py. This replaces the
cloud/ monolith for pretraining. She wrote it to learn, not to delegate —
do not rewrite it wholesale.

chars/token, trained on clean Gutenberg only so BL's damaged words never
enter the vocabulary. A 32k variant was measured and rejected. All tokenizers
moved out of data/ into prose/tokenizers/.

2048, 245.2M params, batch 48, --compile, 50,000 iters = 4.92B tokens =
4.9 epochs, 388.9 min at 211K tok/s avg, $16.86. Final train 2.6699 /
val 2.7814** (val ppl 16.2).

overfitting across 4.9 epochs, so dropout 0.1 was the right call. The
train/val gap widened 0.00 -> 0.10 only because train fell faster; gap
width alone is the wrong test, val direction is the right one.

end (some of that is the LR anneal, not pure learning). More tokens is a
live option for v12, not a dead end.

keys (the raw_model save works), generates grammatical Victorian prose
with correct dialogue punctuation and split attribution. Main flaw is
repetition loops** ("disturbed and disturbed") — seen only at temp 1.0 /
top_k 40, the test_generation.py defaults, so not yet separated from
sampler artifact.

with no chat shape first. SFT is the untouched next lever.

step_25000 (an epoch-arc trio for comparing voice over training). The
other six died with the instance.

v8's anneal taught the chat FORMAT and SFT taught the VOICE; v11 skipped the
anneal on purpose to see the raw base, so the format stage is what's missing.
Hand-graded pool is 781 convos / 3,050 assistant turns / 71k supervised
tokens (graded_conversations 343 + continuous_conversations 438 — she treats
these as the SAME data, one fixed-3-turn and one variable; build_sft_dataset
already agrees, weighting continuous_short at 5 like graded). Avg reply 23
tokens = the 1-2 sentence target. Thin for teaching format AND voice at once,
hence the anneal.

gutenberg_fetch_7.py pulled 526 drama texts -> 195,039 speaker-turns
across 532 plays, 7,113 play-scoped speakers, ~4.78M tokens
(after excluding 14 verse-drama files) (vs 55,654 in chat_named). Why plays:
they need NO attribution inference — the speaker is printed — so the
tiny-prose-dialogue-extraction-bug stays off the critical path entirely.
Median turn 9 words, 86% under 25. Male-authored is fine: the anneal tilts on
SPEAKER gender, and plays label speakers explicitly, so voice_tag.py works
better here than on prose.

characters (servants, villagers): in the Galsworthy compilation 10 of 143
speakers held it, and dropping everyone above 8% apostrophe-elision cost
3.7% of turns vs 14,326 for rejecting the file. Elision% is the specific
signal; OOV% false-positives on proper nouns.

names merge across plays otherwise), run voice_tag, apply the ~7% elision
gate. Judgment calls flagged by the script: 4 verse-drama files (Tennyson,
Yeats x2, Boker) and Synge's Irish dialect.

~1% of pretrain compute, so 3-5 epochs over an ~8M-token corpus is ~4 min /
~$0.15; rent an hour and sweep mixes rather than grinding on the Mac).**

turns are in the anneal, SFT val measures recall, not generalisation.

tiny-prose-dialogue-extraction-bug.

with a 17% tail over 40 words. Plays sit on the 1-2 sentence target;
book dialogue supplies range. Default mix ~50/50 BY TOKENS (plays are only
4.78M vs chat_named's 13.3M, so plays need ~3.5x weight), then sweep.

48,452 female-spoken play turns across 438 plays / 1,371 characters
(56.2% of gendered turns, ~1.08M tokens — better than v10's 42% prose tilt). Hold out
whole PLAYS, not turns, or the anneal has seen the surrounding context.

(12.3%) open with an orphaned answer — 2,536 of them female-spoken, vs a
total hand-graded pool of 781. The reply survived and only the cue is
missing, which is exactly the half she writes. See
feedback-never-filter-on-the-cue.

— each was SILENT and each cost real data:** three speaker layouts, not two
(NAME. text, NAME\ntext, and NAME\n\ntext — the third alone hid 68 real
plays / 56,327 turns, nearly all the Pinero, Maugham, Grundy and Shaw's
Candida, i.e. exactly the drawing-room comedy the anneal wants); stage
directions that WRAP ACROSS LINES (Barrie) which need stripping at file
level before line-splitting; honorific+initial (MRS. L.) and parenthetical
delivery notes (MRS. CLANDON (with quiet force).) both collapsing speakers
to bare MRS; multi-line speeches truncated to their first typeset line;
cast lists and tables of contents read as turns; four spellings of ellipsis.

Ranks extremes well (Browning 0.43, Galsworthy 0.00) but the middle is muddy
— The Importance of Being Earnest scores 0.25 and The Melting Pot 0.29,
both prose, because layout-B files inflate the score. 12 files hand-picked
for exclusion at >0.28 (Browning, Shaw's Bashville, Wilde's Florentine
Tragedy, both Tennysons, both Yeats, Rice, Boker, Crowley, Housman's
Alcestis, Phillips' Nero, Binyon's Arthur, and Oscar Fay Adams' A Motley
Jest — Victorian Shakespeare pastiche). WIRED IN as VERSE_EXCLUDE.

batch 64, lr 1e-4, ~3.8 min each on a GH200 at 217K tok/s, ~$2 total.
Replies drop from paragraphs of narrative to 1-2 sentences — that was
the whole question and the answer is yes.

SFT teaches voice). B = plays x3 (20% plays) is the pick; A (8%) close
second. c_plays_x8 already recites play lines (Mr. Malone from John
Bull's Other Island); d_plays_only memorised outright (train 0.27 / val
3.58 on 1.5M tokens seen 30x).

variant's val comes from its own corpus, and a play-heavy corpus is more
predictable. v11 pretrain val 2.7814 vs anneal val ~2.57 means nothing.
Judge by GENERATION — mkm_train/sample_anneal.py runs fixed prompts
across every checkpoint.

stop convention exists in the corpus. That is SFT's job, not the anneal's.

is cheaper than storing — the 3.7GB download took longer than the training.

--resume restores the step count so resuming a finished 50k run at
--max-iters 50000 trains nothing), pos_emb slicing (2048 -> 512, an anneal
wants a shorter context and pos_emb is sized to the block), and it now
DROPS zero-weight data sources (otherwise blbooks still loaded and val
windows came from pretrain prose while training on chat).

of narrative and stitched unrelated moments). convs 226,866 -> 266,801,
turns -10%, avg len 7.2 -> 5.5. Book exchanges 97,950 -> 76,409;
book_pool_sft_top 6,788 -> 4,666. Worth it: 42% of the old reviewed cues
no longer resolve, and a visible chunk of those were the broken pairs.

id. 337 of 580 survived.

19% -> 2% once three prompt rules landed, all found by READING pairs:
quoted phrase must survive, anaphoric hooks ("so do I") need their verb,
and don't write the reply. All three are the same shape — the reply
depends on something in the cue and the rewrite dropped it.

Llama-3.2-3B matches Laguna here (~100x faster) once the prompt is right —
it failed the REWRITE task only because that needs two-text discrimination.

extracted exchanges, 23% on invented dangler cues. That 8x gap is the
real evidence it reads for responsiveness. 1,446 dangler survivors.

the prompt produced the SAME cue for three different orphans; a tuned
prompt produced "I've been meaning to ask you" 4x in 20. NEGATIVE examples
are safe to show, POSITIVE ones get copied. opener_report() now flags it
automatically and strip_filler() removes filler openings — but only when
what remains stands alone as a sentence ("I've been trying to reach her"
is content, not filler).

not provenance — she invents when writing cues by hand too.

NO" to YES took the veto rate to 0%.

mlx scripts now check free memory first.

load an mkm_train checkpoint — different pos_emb/head naming). Loss masked
to assistant tokens. On the 781 graded convos both anneal variants peak at
step 80, val 2.118 (A) vs 2.122 (B) — indistinguishable, so the anneal
choice washes out through SFT. B still preferred on generation (takes more
initiative: "But what's it like?" vs A's "Ah!").

120 samples/setting: median 14 words, nothing over 40, 0% loops. A
REPETITION PENALTY MAKES REPLIES LONGER, not better — it pushes the model
away from tokens it has used, including the ones that end a reply. The
ghost-loop was one unlucky draw, not systematic.

not a training bug ("I have no doubt I do. I cannot believe in them."). It
survived pretrain, anneal and SFT. One-sentence replies are structurally
immune to it — an argument for short output beyond just hitting the target.

modernise_cues.py + review_cues.py. Laguna, thinking OFF (it's register
transfer, not reasoning; with thinking on the scratchpad leaks). Projects to
~7,945 pairs vs 781 hand-graded.

Inventing produces a question that fits any answer — manufactured
coherence. The exchange already HAS a real question; modernising it keeps
the semantic relationship real.

rules made it copy-edit instead of translate; leading with "write what
they WOULD have typed" + worked examples fixed it.

model left alone writes prompt-ese. Her EDITS are the best examples.

the reply instead. Laguna handles it fine and is fast (8B active).

~1,500 rewritten + ~1,000 danglers + some plays.

triple-quoted string in voice_tag.py put a, the, and into the name
set, so Speaker A resolved FEMALE. Inflated the dangler female count
13,454 -> real 2,844. Pools were unaffected (books guard placeholders
explicitly). Caught because the number looked TOO GOOD.

breaks (max_gap_chars=250 spans ~40 words of narrative), and they read fine
unless you check the reply. I costed the judge against all 6,788 book
exchanges (13h); the mix only needs ~1,500 = <3h overnight. Run it BEFORE
reviewing, so there is less to read — not instead of reading.

questions with checkable answers ("is turn B a reply to turn A"), never
"is this good prose" — selecting on a modern model's taste would import that
taste. judge_calibrate.py measures the judge before trusting it, using
plays as free ground truth: POSITIVE = source-adjacent turns, NEG_STITCH =
two plays spliced, NEG_DISTANT = same play far apart. NEG_DISTANT accuracy
is the number that decides it. Judge PAIRS not whole convos, so a bad seam
trims a conversation instead of discarding it.

Llama-3.2-3B: 70.0% vs a 66.7% always-NO baseline, 30% recall — a refusal
reflex, not judgment. Laguna S 2.1 (thinking off, "if unsure say NO"):
63.3% vs 66.7% baseline, POSITIVE 85% / NEG_STITCH 60% / NEG_DISTANT 45%.
NEG_STITCH 60% is the tell — two lines from entirely different plays,
and it called 40% of them real exchanges. If the easy case fails, the hard
case is worthless. ALWAYS print the always-one-answer baseline next to
accuracy; both models looked plausible until compared against it.

off — first truncated at 4 tokens, second disabled) and give the judge the
SURROUNDING TURNS for context, not two isolated lines.** GLM-5.2 rejected:
365GB against 128GB of RAM, it would page constantly.

gaps find breaks mechanically (median gap 3 lines, p90 8; >10 lines flags
4.5% as scene breaks). The judge gates chat_named ONLY, so a plays-only
anneal can proceed while the judge question stays open.

female holdout from verified conversations — not the other way round.

gets a str, wants a class). judge_calibrate.py carries a shim rather than
pinning transformers in the shared venv. Models live in
/Users/melissa/models/hf-cache; use venv/bin/python for anything mlx.
Laguna also needs its own laguna.py in site-packages/mlx_lm/models/,
wiped by every mlx-lm upgrade ("Model type laguna not supported"); the
file ships in the model snapshot and judge_calibrate.py now reinstalls it
automatically.

turns (chat_named is Gutenberg-derived). Too dirty for SFT by the two-bars
rule, but the ANNEAL bar is lower, so it's the volume option if plays aren't
enough.

lambda-workflow (GH200 now the default instance, per-minute billing).

Current state (2026-07-24) — superseded above

The iterative loop (settled workflow)

  1. Grade conversations interactively vs latest best.pt (cloud/grade_conversations.py → prose/json/graded_conversations.jsonl; provenance + gen_temp tracked)
  2. scripts/build_sft_dataset.py --chat-unattr-weight 1 — rebuild-from-scratch mixing (data isn't "used up"; each SFT round starts from v8 base which never saw graded data)
  3. cloud/sft.py --resume prose/models/model_v8/final.pt — short run, take best.pt
  4. Repeat. Retrain every ~200 new graded convos (compounding: better base → higher keep rate → faster grading)

Goal: ~1,000 hand-graded convos (~3,000 approved turns; LIMA-style reasoning). Milestones: 300 = graded dominates signal; 600 = can drop graded weights to 1-2; 1,000 = reassess. Diversity > count (vary prompt types, turn depth 2-4, emotional registers). Plan: keep ~10 fixed benchmark prompts never trained on, for round-over-round comparison (not yet picked).

Hard-won pipeline gotchas

Related: tiny-prose-goals-constraints (NO synthetic data; 1-2 sentence reply target), feedback-minimal-change, dev-hardware.

Linked from