Tiny prose corpus retrieval
Melissa's unease with the w) write-your-own box (2026-08-08): "I still feel a
little like writing is false... maybe writing by hand is TOO MUCH of my hand
behind the curtain?" — plus the plain craft problem that composing convincing
Victorian prose from scratch is hard.
She had already been working around it by grepping the raw books and rephrasing
hits by hand. That instinct is the resolution, and it now has a tool:prose/scripts/quote_index.py indexes every quoted utterance inprose/data/gutenberg/ (~456k lines, 650 books) into a sqlite FTS5 DB, and the
grader's c) panel searches it per-turn and drops a line into w).
Her actual loop (clarified 2026-08-14 — do not misread this): she types a
MODERN user turn, checks the c) results, and if nothing matches she re-runs
the search with more Victorian phrasing. The Victorian wording is a query
device only — the turn that goes into training stays modern, deliberately,
because that is how she'll really talk to the model. quote_index.py now
supports FTS5 phrase search (text:"that is not my name", optionaltext:/cue:/before:/after: scope) which is exactly the step-3 tool.
Two design facts worth keeping:
- Proper nouns are stripped from queries by default. Searching a false-name
probe with the name returns lines about that character; without it you get
the conversational move (people correcting a misidentification), which is
what's actually needed. names=1 / the "names" checkbox restores them.
- Turns pulled this way are logged with
provenance: "corpus"and a
corpus_source record, distinct from "written" — so the split between a
real author's words and her own hand stays measurable.
Why: the no-synthetic-data rule bars a second undisclosed intelligence in
the channel, not the medium's own hand — but she'd rather lean on dead authors
than on herself, and retrieval satisfies the rule more strictly than editing
model output does.
It works, measured (2026-08-12). sft8 (trained after the corpus panel was
in heavy use — 41% of recent turns) beat sft6 on 4 of 4 blind-marked prompts,
despite being trained 198 steps to sft6's 500. A follow-up best-vs-final
comparison on sft8 came out a dead tie with 40% byte-identical samples, which
rules out lucky checkpoint selection as the explanation.
How to apply: when she's stuck for a reply or discouraged about writing,
point at the corpus panel rather than urging her to compose. Related:
tiny-prose-goals-constraints, tiny-prose-side-project.