How much does it cost to pretrain a modern LLM?
hardAnswer
- Rule of thumb: 6*N*T FLOPs, where N = parameters and T = training tokens.
- LLaMA-3 8B on 15T tokens = ~7.2e23 FLOPs.
- On an H100 doing ~1e15 FLOPs/s in bf16 (~50% MFU), that's ~1.5M GPU-hours (~60-100M).
- Frontier training is 10-100x more expensive than fine-tuning; inference at scale often dominates lifetime cost.
Check yourself — multiple choice
- A few thousand dollars
- 6·N·T FLOPs → LLaMA-3-8B ~ 60-100M; inference at scale often dominates
- Free with public GPUs
- Independent of data size
Training cost ~ 6·N·T; frontier LLMs cost 100M+ in compute alone.
#pretraining#cost#production
Practise LLMs & GenAI
214 interview questions in this topic.
Related questions
- What's the practical minimum tokens-per-parameter for a competitive LLM?
- How should max_tokens be set in production?
- What is prompt caching and when does it help?
- How do you use one prompt to extract multiple fields at once vs many separate prompts?
- What can you cache in a RAG pipeline?
- How do you pick a GPU for serving a 70B LLM?