Scaling Plan — Phase 2

After the May 2026 experiments (see LEARNINGS.md), the path forward is clear:
more data, smarter data, then bigger model — in that order.

This document is the strategic plan I (Melissa) can refer back to when ready
to move beyond the current 12M from-scratch baseline.

---

The starting point

• Current model: 12M params (L6, E384), trained on ~50K pairs + ~1K multi-turn
• Ceiling hit: stylistically nailed, conversationally limited
• Confirmed dead ends: full fine-tune (catastrophic forgetting), LoRA on GPT-2 (wrong base), LoRA on DialoGPT-small (right idea but too small/wrong-flavor)
• Production answer for now: Claude API at /han route on ocean-chat-simple
• Long-term: keep building toward a tiny model that can actually converse

---

Five dimensions of the plan

1. Data sources to mix

The goal is diversity at scale, not just volume. Stylistic narrowness was
exactly the problem with the last "more data" attempt.

Combined target: ~150–200K total pairs with conscious balance.

2. Synthetic generation via Claude — cost math

Sonnet 4.6 pricing: ~$3/MTok input, $15/MTok output.

Frugal strategy: generate bulk with Haiku, have Sonnet grade and filter to
keep only the top ~30%. Best of both — cheap volume + high quality.

Validated 2026-06-13: A/B-tested Haiku 4.5 vs Sonnet 4.6 on the reply suggestions in the assist-mode collector (claude-haiku-4-5). Haiku held Han's terse voice well — close enough to Sonnet that it's the right default for bulk generation. Keep Sonnet for the Speaker-A openers and for grading; let Haiku carry the volume. Switchable in the local collector via the "replies:" dropdown.

3. Why Claude-synthetic will dramatically beat the local 14B/8B noir attempts

• Maintains scene coherence across many turns (local models lost it by turn 3)
• Follows style rules tightly (no stage directions, no quotes, terse)
• Real variety when prompt is parameterized (settings, characters, registers)
• No CJK injection leakage to filter out

4. Model size to match the new data

Rough rule: ~1M params per 10K training pairs gives a healthy regime
(small enough to not overfit, big enough to absorb the signal).

5. When to leave the laptop

Up to ~50M params trains acceptably on M-series MPS in hours.
At 100M+, want a rented GPU.

Cloud options:

• Lambda Labs, RunPod, or vast.ai for A100/H100 access at $1–3/hour
• Full training run at 100M params: ~5–10 hours = $10–30 per run

---

Next deliverable when ready

A claude_synthetic_pairs.py script that:

1. Takes a parameterized prompt template (scene, mood, character relationship,

register)

1. Hits Claude API with batched generation (10 pairs per call to amortize overhead)
2. Validates output (length, format, banned patterns, CJK)
3. Appends to data/claude_synthetic_pairs.jsonl
4. Tracks running cost in real-time so I can stop when I hit my budget

---

Order of operations

1. Now → next few weeks: chat with /han casually, let han_dialogue.jsonl

accumulate. Periodically run extract_pairs.py and check the mix.

1. When the natural data feels rich enough (~5K+ pairs): start the synthetic

Claude pipeline. Budget ~$50 first to test quality before going bigger.

1. Rebuild combined.jsonl with balanced mix from all sources.
2. Train at 20–30M params (still on Mac) and measure: did the conversational

capacity actually improve, or did I just overfit harder?

1. Visualize attention on the new model with viz_attention.py to compare

against the 12M baseline. Look for: are more heads doing genuine work? Is
the attention-sink ratio in a healthy range?

1. Only then consider 50M+ in the cloud — if the 20–30M run validates the

scaling intuition.

---

What "done" looks like

A tiny model (≤50M params) that:

• Holds character across 3+ turns
• Responds appropriately to in-distribution AND novel casual input
• Generates lines I'd actually be happy to keep
• Trained for under $200 total in API + compute costs

If I get there, I've proven the thesis that a small from-scratch model
CAN be conversational, given enough thoughtfully-collected data.

If I don't, I've still learned more about transformers than 95% of people
who use them. Worth doing either way.