Modal stack baseline

Modal-based stack for Qwen2.5-7B LoRA fine-tuning + scale-to-zero inference. Built June 2026, validated end-to-end. Lives in cloud/ in the tiny-chat repo:

• cloud/requirements.txt — CUDA-side deps (baked into Modal image, NOT installed on Mac)
• cloud/train_lora.py — transformers + peft + trl SFTTrainer (port of mlx_lm.lora)
• cloud/app.py — Modal app: @app.function train + @app.cls HanModel with @modal.asgi_app() FastAPI

Operational numbers (Qwen2.5-7B-Instruct, ~2k training pairs, A10G):

• Training: ~12.5 min for max-steps 600, ~$0.23 ($1.10/hr A10G)
• Image first build: ~3-5 min (cached after)
• Base model first download: ~1-3 min into HF cache on the volume (cached after; HF_TOKEN speeds this up but isn't required)
• Inference cold start: ~30-60s (loads 7B + adapter from volume)
• Inference warm: ~1s per candidate, best-of-3 in ~3s
• Idle cost: $0 (scale-to-zero via scaledown_window=300)

Deployed endpoint URL pattern: https://<workspace>--<app-name>-<class-name>-<method>.modal.run (current: beesandtrees--han-qwen7b-hanmodel-web.modal.run). Workspace is your Modal account slug.

Gotchas hit on first build (all fixed in the current cloud/ files, but watch for these if reusing the pattern):

1. SFTConfig API drift in trl 0.15+. max_seq_length= was dropped from SFTConfig and throws TypeError: unexpected keyword argument 'max_seq_length'. Just remove it; let the tokenizer's default truncation handle it. Short chat data doesn't need explicit limits anyway.

1. fastapi and pydantic must be in cloud/requirements.txt explicitly. They're not transitive deps of transformers/peft/trl on a clean Debian image. @modal.asgi_app() will fail at deploy time with ImportError: No module named 'fastapi' if you forget.

1. HF_HUB_ENABLE_HF_TRANSFER is deprecated by huggingface_hub. Use HF_XET_HIGH_PERFORMANCE=1 instead, and drop hf_transfer from requirements (no longer used).

1. transformers deprecated torch_dtype= for dtype= in from_pretrained(). Still works as warning, but use dtype=torch.bfloat16 in new code.

1. .env → modal.Secret.from_dotenv() is the idiomatic way to forward local env vars (HF_TOKEN, ANTHROPIC_API_KEY) into Modal containers without baking them into the image. Mount on each @app.function and @app.cls that needs them.

1. Volume caching pattern: set HF_HOME=/vol/hf_cache via image.env({...}) so the 14GB base model downloads once and is reused across every train and inference container. Massive cost saver.

1. modal run is tethered — use --detach for long jobs. modal run cloud/app.py::train streams logs over the local client; if the laptop sleeps/disconnects, the remote function gets killed mid-run. Discovered the painful way on a multi-hour training run. For anything more than a few minutes, use modal run --detach <fn> so the function survives independently. Watch logs after with modal app logs <app-name> or the dashboard.

1. subprocess.run(["python", ...]) needs -u (or PYTHONUNBUFFERED=1) for live logs. Python switches stdout to 4KB block-buffering when not on a TTY, so prints from a launched script don't appear in Modal logs until ~20+ prints have accumulated — looks "stuck" for many minutes. C/C++ logs (e.g. sentencepiece) flow fine because they use stderr. Always pass -u to subprocessed Python.

How to apply: When the user wants a new fine-tune iteration, edit cloud/train_lora.py for hyperparameters, run modal run cloud/app.py::train, then modal deploy cloud/app.py to swap the inference endpoint. Both commands are user-invoked per feedback-billable-commands. For an entirely new model/adapter, version the output dir (e.g. /vol/adapters/han_qwen7b_v2) so prior adapters stay accessible.