tiny-chat roadmap

Long-running side project to train a sardonic-detective tiny LM from scratch
and learn ML mechanics along the way.

Guiding constraints:

• Stay on a MacBook (no waiting for Mac Studio)
• Long-running, low-pressure pace
• Learn the mechanism, not just ship a thing
• Use Claude API sparingly but strategically (currently <$1 spent)

Current state (May 2026): 8M-param decoder-only transformer trained on
~40K pairs (mostly Ollama-augmented + ~340 hand-curated). Model has personality
and answers simple greetings; longer-form output gets noisy. Loss curves show
classic overfitting past iter 2500 — capacity exceeds data.

---

Phase 1 — Data ramp (this week + next)

Goal: more high-signal training data without spending forever generating it.

1a. Continue hand-curated pair collection — IN PROGRESS (~700/1500)

• Now collected via tiny-chat/web/train (migrated from ocean-chat-simple)
• Target: ~1500 total hand-curated pairs (currently ~700, ahead of schedule)
• Why it matters: hand-curated pairs are roughly 5-10x denser in identity

signal than synthetic. Even at 3-5% of the total corpus they tilt the model's
voice meaningfully.

1b. Save full conversation transcripts — DONE

• /train saves to data/conversation/melissa_conversations.jsonl on "end & start new"
• generate_dialogue.py saves to data/conversation/two_model_conversations.jsonl
• Why it matters: transcripts preserve conversational flow. Phase 3

multi-turn training feeds on these.

1c. Two-model Ollama dialogue generation — ACTIVE (replaces Claude API plan)

• generate_dialogue.py runs two Ollama models alternating as Speaker A / B
• Currently qwen2.5:14b + llama3.1:8b — the cross-model interaction

produces more dynamic dialogue than any single model can

• Saves both transcripts AND extracted (A→B and B→A) pairs
• Has CJK-filter to drop qwen prompt-injection attempts
• Why this beat the Claude-API plan: quality at 14B is good enough that

paying for Claude API for bulk gen isn't worth it. The two-model setup also
generates Phase 3 multi-turn data for free as a side effect.

• Target: 500+ conversations → ~2500+ multi-turn pairs

1d. More glue from qwen2.5:14b — ACTIVE

• Conversational primitives (greetings, acks, refusals) using generate_glue.py
• These don't need to be heavily noir-flavored; even Bogart says "hello"
• Target: 3000+ glue pairs at qwen quality (already at this point)

---

Phase 2 — Tooling (concurrent with Phase 1)

Goal: stop hand-running everything; make experimentation cheap.

2a. Eval script with fixed prompts — PARTIAL (integrated in web UI)

• The 10 canonical prompts on the web UI auto-run when you load a checkpoint.

Standalone evaluate.py for CLI use isn't built yet but probably not needed.

• Why it matters: "vibe check" doesn't scale to multiple runs. A fixed

prompt set gives you apples-to-apples comparisons of how a change affects
the model.

2b. A/B compare script — TODO

• Either a CLI compare.py <run_a> <run_b> OR a web UI mode that loads two

checkpoints and shows responses side-by-side

• Why it matters: seeing "model with 3K hand-curated pairs" vs "model with

500 hand-curated pairs" at a glance teaches you which changes actually moved
the needle.

2c. Web UI — DONE

• tiny-chat/web/main.py runs on port 8001
• / for chat with any saved checkpoint, /train for pair collection
• Dropdown switches checkpoints, canonical prompts auto-run.

---

Phase 3 — Multi-turn capability (weeks 3-5)

Goal: the bot remembers context from earlier in the conversation.

3a. Multi-turn data generation — DONE (via two-model dialogue)

• generate_dialogue.py produces multi-turn transcripts as a side effect
• Currently ~1000 transcripts (5500+ turns) in data/conversation/

3b. Update training to use multi-turn data — DONE

• train.py now loads both pair sources and multi-turn transcripts as

unified "blocks". Pairs = 2-element blocks, transcripts = N-element blocks.

• Verified improvement vs single-turn-only training:
• best val loss 2.33 → 1.62 (-30%)
• val minimum moved from iter 2500 → 10000 (4x more learning room)
• train/val gap 2.50 → 1.43 (less overfitting)

3c. Update chat.py + web UI to feed conversation history — TODO

• Currently: each user message → standalone single-turn prompt
• New: maintain conversation history client-side, feed last N turns as context
• Why it matters: even though the model NOW learned multi-turn patterns,

we're still feeding it single-turn prompts at inference. The capability is
trained but un-activated. This is the unlock.

---

Phase 4 — Model scaling (week 5-6+)

Goal: use the bigger corpus productively, push past current ceiling.

4a. Scale model to ~15-20M params

• N_LAYER 6→8, N_EMBD 384→512 (~18M params)
• Only worth doing AFTER Phase 1 puts you north of 50K pairs — otherwise the

current overfitting just gets worse

• Why it matters: the val-loss curve showed train and val diverging at

iter 2500 — that's "model can memorize all the data." More params on the
same data = more memorization. More params + 3x more data = more capability.

4b. Sampling improvements

• Add top-k filtering to generate() (currently pure multinomial)
• Add repetition penalty
• Tune temperature per-character (lower for short prompts, higher for long)
• Why it matters: these are inference-time changes — same trained model,

different output behavior. Cheap to experiment with.

4c. Learning rate scheduler

• Try cosine decay or linear warmup
• Most "real" LM training uses one of these
• Why it matters: constant LR is fine for tiny experiments; bigger models

benefit from schedules. Educational to see if it actually helps at your scale.

---

Phase 5 — Stretch / future

Things to consider once the core feels good:

• Distillation: train your tiny model on logits from Claude (not just text)
• Quantization: shrink the saved model for distribution
• Compare BPE vs sentencepiece-unigram vs char-level on identical training
• Mix in a tiny amount of TinyStories-style data for coherence baseline
• Build a leaderboard of your own runs with val loss + eval-prompt outputs

Knowledge insertion — give the bot Melissa's world model

Bake in personal/factual knowledge so the bot can answer questions about
things like favorite movies, daily routines, opinions, places, people —
all delivered in detective voice. Riffs on the HalBot tradition where a
bot's personality includes a specific knowledge bias.

• New /knowledge route in the web app: bot asks Melissa factual/personal

questions across categories (movies, food, places, opinions, routines).
Melissa answers in-character. Each (Q, A) saves as a training pair.

• Tag knowledge pairs with category metadata so we can up-weight them.
• Mix at ~5-10% of total training data.
• Reality check at this scale: the model can memorize ~hundreds of

specific Q→A mappings as lookups, but won't generalize across phrasings
("favorite movie?" ≠ "favorite film?"). For tight prompt matching it works;
for free-form questioning it'll often miss. That's the same memorization-
not-generalization ceiling we keep hitting.

• Why it's fun anyway: answers come out in noir voice, so you get a

detective who happens to know Melissa's opinions on things. Unique flavor.

---

Open questions / things to revisit

• At what scale does multi-turn context actually emerge as reliable behavior

(vs. faint signal)? May need 20M+ params + 5K+ multi-turn dialogues.

• Is the current BLOCK_SIZE=128 actually limiting anything? Cheap to check

with a single run at 256.

• Should "Claude" pairs be filtered/dedupe before merging into combined.jsonl?

Probably yes; need a small quality script.

---

Done so far

• Char-level → BPE tokenizer migration
• Filter Cornell to single character (Han Solo)
• Filter Cornell to thematic archetype (Bogart-lineage tough guys, 824 pairs)
• Synthetic generation via Ollama 3B then 8B
• Pair augmentation via Ollama with seed-based topic variety
• Hand-curated pair collection via Claude-as-Speaker-A interface
• Loss/curve logging + CSV/PNG per run
• Refactored Cornell loader out of train.py
• Model save/load to enable replay
• Identified overfitting via val-loss curve analysis