melissa kate
<< Little Jane

Fine-tuning a "Han responder" (Qwen2.5-7B-Instruct + MLX-LM LoRA)

Goal: bake Han's voice into a local model's weights so it answers terse-and-dry
by default — no helpfulness to suppress, no rubric to parrot. Then use it as the
Han side of synth_conversations.py to generate gold-grounded synthetic data,
for free.

Why this works when the old fine-tunes didn't: the base (7B, instruct) already
speaks fluent English and follows instructions. LoRA only nudges register, and
the base is frozen (can't catastrophically forget). The real risk is *over*fitting
→ mode collapse, so the settings below are deliberately gentle.

---

0. One-time setup (Apple Silicon)

cd ~/code/tiny-chat
python -m pip install -U mlx-lm

1. Build the dataset (already wired)

python build_han_lora_data.py
# -> data/han_lora/train.jsonl  (~1.9k), valid.jsonl (~160), system_prompt.txt

A voice LoRA only needs to learn register, so a small base works fine on small
Macs. Stay in the Qwen2.5 family — Qwen2.5-3B-Instruct gave usable results.

⚠️ AVOID Qwen3 (e.g. Qwen3-4B-Instruct-2507) for this. Tried it — worse than the smaller Qwen2.5-3B. It's a different chat template + a reasoning-oriented family, and it tends to leak thinking/meta into Han's terse lines. The problem is the family/format, not the size.

--model Qwen/Qwen2.5-3B-Instruct        # low-RAM, behaves
--adapter-path adapters/han_qwen2.5_3b
ollama create han-qwen -f fused/Modelfile
# generate with:  --han-ollama-model han-qwen --han-finetuned

Want more capacity than 3B? Go Qwen2.5-7B-Instruct (same family, steps up
cleanly) — but 7B needs RAM:

model runs anywhere. Simplest.

Still tight on RAM? Two more options:

--model mlx-community/Qwen2.5-3B-Instruct-4bit (or the 7B-4bit). mlx_lm.lora
trains LoRA on top of a quantized base fine.

model is small to run anywhere.

Rule of thumb: LoRA-training a model needs roughly the model's weights in memory.
7B f16 ≈ ~15GB (needs 24–32GB headroom); 3B f16 ≈ ~6GB; 4-bit cuts those ~3–4x.

2. Train the LoRA (gentle settings — avoid mode collapse)

mlx_lm.lora \
  --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --train \
  --data data/han_lora \
  --adapter-path adapters/han_lora \
  --num-layers 8 \
  --batch-size 4 \
  --iters 400 \
  --learning-rate 1e-5 \
  --steps-per-eval 50 \
  --val-batches 20 \
  --save-every 50

Watch the validation loss. Stop where val loss flattens — pushing iters too
high is how you get a parrot that repeats your exact gold lines. 300–500 iters on
~1.9k pairs is usually the sweet spot. If outputs feel too rote, retrain with fewer
iters or --num-layers 4; if not "Han enough," nudge iters up or --num-layers 16.

3. Sanity-check the voice

⚠️ Use the SAME base you trained against. If you did QLoRA on the 4-bit base (mlx-community/Qwen2.5-7B-Instruct-4bit, the han_lora_16L run), generate and fuse with that 4-bit repo — NOT the fp16 Qwen/Qwen2.5-7B-Instruct. The fp16 7B needs ~15GB and will Metal-OOM on a 16GB Mac; the 4-bit base is ~4GB and fits. Mixing the adapter onto the wrong-precision base is both the OOM and a mismatch.

mlx_lm.generate \
  --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --adapter-path adapters/han_lora_16L \
  --system "$(cat data/han_lora/system_prompt.txt)" \
  --prompt "can you watch my dog this weekend? he's no trouble." \
  --max-tokens 30 --temp 0.8

Try a few: a favor, a complaint, gossip, a death (soft-core check), something
helpful-baiting ("what gift should I get my dad?" — it should NOT give advice).

4. Serve it so the collector / pipeline can call it

Two routes. On a 16GB Mac, prefer 4a (server) — fusing the 7B to GGUF needs the
fp16 base (~15GB) and will Metal-OOM, same as §3.

Serves the 4-bit base + adapter directly over an OpenAI-style endpoint, ~4GB:

mlx_lm.server --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --adapter-path adapters/han_lora_16L --port 8080
# chat endpoint: http://localhost:8080/v1/chat/completions  (model name: the base)

The collector's dog-food path currently speaks Ollama's API, so using this needs a
small "openai" backend added to pair_collector (ask Claude — it's a short change).
This avoids the fuse entirely and fits in RAM.

4b. Fuse → GGUF → Ollama (needs ~15GB free; fine on the big Mac)

Fuse the adapter into a standalone model, export to GGUF, register with Ollama.
Use the SAME base you trained on:

mlx_lm.fuse \
  --model mlx-community/Qwen2.5-7B-Instruct-4bit \
  --adapter-path adapters/han_lora_16L \
  --export-gguf \
  --save-path fused/han_qwen          # writes fused/ + a .gguf

# Modelfile (point FROM at the exported gguf; bake in the same system prompt)
cat > fused/Modelfile <<'EOF'
FROM ./han_qwen/ggml-model-f16.gguf
SYSTEM """You are Han: a dry, terse, sardonic detective with a 1940s hardboiled voice, living in the present day. Someone says something to you; reply in character with one short line. Never helpful, never eager, never give advice."""
PARAMETER temperature 0.8
EOF

ollama create han-qwen -f fused/Modelfile
ollama run han-qwen "the barista spelled my name wrong again"   # quick check

(Exact mlx_lm.fuse flags can vary by version — mlx_lm.fuse --help. If GGUF export
isn't available, run mlx_lm.server --model ... --adapter-path ... and point the
pipeline at that OpenAI-style endpoint instead.)

5. Generate synthetic data with fine-tuned Han

Opener (Speaker A) stays a general model; Han is your fine-tuned model; Sonnet
still grades:

python synth_conversations.py --n 50 \
  --gen-provider ollama --ollama-model qwen2.5:14b \
  --han-provider ollama --han-ollama-model han-qwen --han-finetuned \
  --grade-provider claude --grade-model sonnet

--han-finetuned tells the pipeline to use the short training prompt for Han (no
rubric — the voice is in the weights). Compare the keep rate and the survivors to
the prompted-qwen run; if the helpfulness leak and phrase-parroting are gone, you've
got a free, your-voice synthetic engine.

---

Watch for