How this calculator works
Enter layers, sequence length, hidden size, batch size. Select Calculate to apply the displayed formula and review the labeled results.
Formula / method
Approximate attention operations = 2 × layers × batch × sequence² × hidden size
Worked example
32 layers, 4,096 tokens, hidden size 4,096 and batch 2 require roughly 4.4 trillion attention operations.
Assumptions and limitations
This uses a dense scaled-dot-product-attention approximation.
Flash attention, sparsity, grouped-query attention, MLP compute and hardware efficiency are not modeled.