tiny-prose running cost ledger
Rough tracking of cloud spend across the project. Update as we go.
Modal (H100 on-demand ~$3.90/hr, per-second billing)
| date | run | cost | notes |
|---|---|---|---|
| 2026-06-30 | v1 pretraining (33M / 11M tokens) | ~$0.80 | first pretraining attempt |
| 2026-06-30 | v2 pretraining (33M / 16M tokens) | ~$1.00 | corpus expansion round 1 |
| 2026-07-01 | v3 pretraining (100M / 395M tokens) | ~$5.60 | BL Books added, big jump |
| 2026-07-01 | wasted runs (aborted tokenizer / stuck encoding) | ~$2-3 | pre-cache-invalidation, before figuring out fix |
| Modal total | ~$10-15 | credits depleted; migrated to Lambda |
Lambda (GH200 $2.29/hr, H100 SXM5 $4.29/hr, H100 PCIe $3.29/hr)
Billing switched to per-minute as of 2026-08-19 (was per-hour).
| date | run | instance | duration | cost | notes |
|---|---|---|---|---|---|
| 2026-07-02 | v4 pretraining (200M / 391M tokens) | H100 SXM5 | 2.25 hr | $9.63 | includes OOM restart + tokenizer + encoding + train + eval |
| 2026-07-03 | Phase 2 SFT (chat fine-tune, 51K pairs) | H100 PCIe | 0.67 hr | $2.19 | best val 2.044 |
| 2026-07-03 | Phase 3 candidate generation (300 prompts × 5) | H100 PCIe | 0.14 hr | $0.47 | for rejection-sampling curation |
| 2026-07-05 | v5 pretraining (200M / 270M words) | H100 PCIe | 1.32 hr | $4.34 | full OCR/inline-marker cleanup pass; final val 3.113 (new tokenizer, not comparable to v4) |
| 2026-08-19 | v11 pretraining (245M / 4.92B tokens) | GH200 480GB | 6.48 hr | $16.86 | 18L/16H/1024, block 2048, batch 48, bf16 autocast + tf32 + torch.compile, 211K tok/s avg; final val 2.7814, no overfit across 4.9 epochs; figure covers benchmark sweep, a relaunch after SSH key loss, and transfer time |
| 2026-08-20 | v11 anneal sweep (4 variants) | GH200 480GB | 0.7 hr | $1.56 | plays-vs-books mix sweep; 4 runs x 3.8 min at 217K tok/s; anneal WORKS — replies drop to 1-2 sentences. (August total $18.42 - $16.86 pretrain.) |
| Lambda total | $35.05 |
Grand total so far: ~$45-50
Upcoming (rough estimates)
- v5 SFT on 51K pairs: ~$2-3
- Phase 3 re-SFT on graded winners + original pairs: ~$2-3