tiny-prose todo

v11 base + anneal done. sft.py works. Remaining work is all DATA.

The pipeline, in order

1. [x] Re-extracted at --max-gap-chars 120 (was 250, which spanned ~40

words of narrative and stitched unrelated moments together).
convs 226,866 -> 266,801, turns -10%, avg length 7.2 -> 5.5.
book_exchanges 97,950 -> 76,409, book_pool_sft_top 6,788 -> 4,666.

1. [x] Re-mapped the reviewed cues — remap_cues.py matches by TEXT, not

id. 337 of 580 survived; the other 42% were pairs the tighter gap
deleted, and a visible chunk of those were the broken ones.

1. [x] Vetoed the pool — 4,666 judged, 132 vetoed (2.8%), **4,534

survive**. Spot-checked vetoes were all genuine breaks.

1. [ ] Rewrite more cues from the survivors — target ~1,500 total.

Batch 3 ran at 98% keep / 2% edit after the prompt rules landed.

1. [~] Danglers — GENERATED AND VETOED, not yet reviewed. 1,877 produced

(Llama-3B, ~100x faster than Laguna and equal quality here), 431 vetoed
(23% — vs 2.8% on real pairs, which is how we know the judge reads for
responsiveness). 1,446 survivors awaiting review.

1. [ ] Plays too — pool_sft_top is 1,320 exchanges, also Victorian on

both sides, also needs cues. Cleaner than books (printed adjacency).

1. [ ] Assemble the mix — 781 gold + ~1,500 rewritten + ~1,000 danglers
2. some plays. NOT all 8,108; diversity beats volume and gold would be

swamped.

1. [ ] SFT on the real dataset, then read what she sounds like.

Banked

• 337 remapped cues + batch 4 in flight + 1,446 vetoed danglers
• 580 cues reviewed across batches 1-3, 99% usable; edit rate 19% -> 2%
• 4 reply filters: too long, too short, fragment, grammatical dialect
• 3 prompt rules: quoted phrase, anaphoric hook, don't-write-the-reply
• veto script, calibrated — DO NOT reword its prompt

Loose ends

• [ ] chat_unattributed has never been used — 80,655 convs / 444,277

turns, every speaker Speaker A/Speaker B. No names means no gender,
so it never entered the pools and nothing reads it. Unusable for SFT
(the reserve is built on female SPEAKERS) but it belongs in the ANNEAL,
where gender is irrelevant and it would add a third to the 331,905-turn
book pool. Excluded by accident, not decision. Low priority: the anneal
had 17.5M tokens and needed ~8-15M, so it was never data-starved —
this is a lever for if SFT disappoints, not a fix.

• [ ] cues_batch1 item 52 mismarked; item 271 has yyo for you
• [ ] split_exchanges.py has no length filter (fixed in modernise_cues only)
• [ ] filter replies that fit ANY cue — "That's too bad, Prosper" teaches no

mapping, so no invented cue can be more correct than another. The mirror
of "a cue that fits any reply is too vague".

• [ ] the Le Fanu exchange scored elision 0.0% despite visible dialect

Settled, don't re-litigate

• Anneal variant B (plays x3). A and B are indistinguishable after SFT.
• Sampling: temp 0.8 / top_k 100 / no repetition penalty.
• Judge is a REJECTOR, never a ranker. Its NO is 80-94% right, its YES is a

coin flip.

• TRANSLATE existing cues; only INVENT for danglers, where the orphaned

answer constrains the question.

• Anneal teaches shape, SFT teaches voice. Gender tilt belongs in SFT.

Backlog

• reply-first sweep inversion
• prose/data/ -> prose/build/ split
• overlap_probable.txt (9), overlap_partial.txt (5)
• prose/notes/review-*.md triage