How this calculator works
Enter transformer layers, hidden size, sequence length, batch size, bytes per value. Select Calculate to apply the displayed formula and review the labeled results.
Formula / method
KV bytes = 2 × layers × hidden size × sequence tokens × batch × bytes per value
Worked example
32 layers, 4,096 hidden size, 8,192 tokens, batch 4 and FP16 need 16 GB of KV cache.
Assumptions and limitations
One key and one value are stored for every hidden feature at the selected byte width.
Grouped-query attention, tensor parallelism, padding, allocator overhead and framework buffers vary by model.