How this calculator works
Enter gpu count, vram per gpu (gib), model weights per gpu (gib), runtime reserve per gpu (gib), kv cache per active request (gib), generated tokens/s per gpu, scheduled utilization (%), fleet availability (%), average output tokens per request, gpu cost per hour ($), target gross margin (%). Select Calculate to apply the displayed formula and review the labeled results.
Formula / method
Worked example
Eight 80 GiB GPUs running 70 GiB model weights, 6 GiB runtime reserve and 2.5 GiB KV cache per request admit 8 concurrent requests; at 120 tokens/s per GPU, 70% utilization and 99.5% availability, they produce roughly 1.76B output tokens each month.
Assumptions and limitations
Weights are replicated on each GPU, KV cache is evenly distributed, throughput is generated-output throughput, and utilization represents productive scheduled time.
Tensor/pipeline parallelism, prefills, batching, queueing, variable sequence lengths, interconnect, CPU/network costs, orchestration, depreciation, taxes and provider discounts are not modeled. Treat capacity as a planning estimate, not a serving SLO.