tiny-chat — project notes / handoff

Context capture so this work is portable across machines (the Cowork chat itself
doesn't sync the local repo — this file does). Last updated 2026-06-22.

What this is

Training a tiny from-scratch transformer to talk like "Han" — a dry, terse,
sardonic 1940s hardboiled detective who now lives in the present day and
reacts to ordinary modern life in that clipped old-school voice. The trained
tiny model is the learning artifact / showcase; Claude API powers the live
/han chat in the meantime.

The voice (what "good" means)

See data/han_voice_rubric.md — the explicit style guide reverse-engineered from
the gold pairs (default register, the rare "soft core" warmth, the wit layer, and
a hard list of what Han would NEVER say — esp. no helpfulness / advice-giving,
no modern tech-speak). It's hand-editable and is what reply-suggestions imitate
and what the synthetic grader scores against.

Data (all in data/)

• melissa_pairs.jsonl — hand-curated gold (prompt → Han reply). ~2,000 pairs. THE asset.
• human_pairs.jsonl — fully hand-written-both-sides gold (from the /human-inputs tool).
• conversation/melissa_conversations.jsonl — full multi-turn transcripts (~284). Teaches turn-taking.
• han_voice_rubric.md — the voice rubric.
• han_lora/ — generated LoRA training set (from build_han_lora_data.py).
• synth_pairs.jsonl / synth_conversations.jsonl — synthetic output (experimental).
• two_model_pairs.jsonl — OLD local-model noir synthetic (period, pre-pivot).
• claude_synthetic_pairs.jsonl — rejected approach (one model invents both sides → hollow zingers).

Collection tools

• Local web/ (FastAPI): /train = assisted collector (Claude opens as a modern

person, you reply as Han; reply-suggestions anchored on rubric+gold; Haiku/Sonnet
selectable; scene auto-suggest). /human-inputs = you write BOTH sides → human_pairs.jsonl.
Run: uvicorn web.main:app --reload --port 8001

• Remote ~/code/ocean-chat-simple (deployed at killerpizza.party): /han mirrors

the assisted collector; /human-inputs ported too. Download data via /download/<name>.

Key decisions (so we don't relitigate)

• Modern-only. Noir scene generation was stripped from the collector; enough noir gold exists. Han KEEPS his 1940s reply voice — modern situations, 40s replies. The anachronism is the point.
• Distribution > volume (LEARNINGS Lesson 1): don't flood with narrow data.
• Haiku holds Han's voice for replies/scenes; Sonnet for openers/grading.
• Worry to respect: don't end up "training a bot to be a subset of another bot." Keep gold the majority; synthetic must be your-voice-grounded + graded.

CURRENT EXPERIMENT — fine-tuned Han generator (in progress)

Idea: LoRA-fine-tune a local model on the gold so Han's voice is in the WEIGHTS
(no helpfulness to suppress, no rubric to parrot), then use it as the Han side of
the turn-by-turn synthetic generator. This is YOUR voice amplified, not Claude's.

Why earlier fine-tunes failed but this won't: those were tiny/wrong-base PRODUCT
models. This is an 8B-class, frozen-base, instruct GENERATOR — the regime where
LoRA reliably works. Main risk = overfit/mode-collapse → watch val loss.

Run order (see FINETUNE_HAN.md for full commands):

1. python build_han_lora_data.py → data/han_lora/{train,valid}.jsonl (~1,889 / 164)
2. pip install mlx-lm; LoRA-train Qwen2.5-7B-Instruct (gentle: ~400 iters, num-layers 8, lr 1e-5). Stop when val loss flattens.
3. Sanity-check generations; fuse → GGUF → ollama create han-qwen.
4. Generate:

python synth_conversations.py --n 50 --gen-provider ollama --ollama-model qwen2.5:14b --han-provider ollama --han-ollama-model han-qwen --han-finetuned --grade-provider claude --grade-model sonnet

1. Compare keep-rate + read survivors vs the prompted-qwen run.

.env (ANTHROPIC_API_KEY) is gitignored — recreate it on each machine; only needed for the Sonnet grader (training + local gen are offline).

Tooling

• find_typos.py — LLM typo scan, --target pairs|conversations, resumes via checkpoint, Ollama default / --provider claude.
• build_han_lora_data.py, synth_conversations.py, FINETUNE_HAN.md — the synthetic/fine-tune stack above.

Open threads / next

• Run the fine-tune experiment; judge keep-rate & voice.
• If good: scale synthetic as a minority of a rebalanced combined.jsonl (gold-heavy + Cornell for conversational baseline), then train 20–30M and measure.
• Backtranslation (rewrite real ordinary dialogue → Han) is an alternative synthetic source if needed.

Idea: best-of-N synthetic with the fine-tuned Han (Melissa, 2026-06-23)

From dog-fooding the fine-tune as the reply-suggester in /train: there's reliably
~1 good reply per group of 3 candidates. So when we build the synthetic
generator, change the flow to exploit that hit rate instead of taking one shot:

1. Local LLM writes the prompt (Speaker A opener) — qwen2.5:14b, free.
2. Fine-tuned Han generates N≈3 candidate replies to that prompt (sample at

temp — the _suggest_replies_finetuned dog-food path in pair_collector.py
already does exactly this).

1. Grader scores the candidates and keeps the BEST one (Sonnet grader, same

"in_voice AND lands" rubric as synth_conversations.py) — pick-best, not
pass/fail. Drop the whole group only if none clear the bar.

Why: single-shot generation throws away the 2-in-3 that miss; best-of-N turns the
same generation into a much higher land rate per saved pair. It's rejection
sampling against the grader — exactly what the by-hand dog-food loop already is
(eyeball 3, take 1).

Implementation note for later: synth_conversations.py already has separate
--han-provider / --grade-provider and a turn-by-turn loop. The change is to the
Han turn (generate N, grade N, emit the argmax) + the grader (return a score, not
just keep/drop). Keep the gold-majority ratio rule regardless.