AI inference: size memory before counting GPUs

Start with the requests users will send

Specify context length, concurrent requests, first-token latency and output tokens per second. A seven-billion-parameter model stored at two bytes per parameter requires about 14 GB of decimal weight storage alone. Runtime buffers, activations, allocator overhead and the KV cache are additional. Quantization changes the weight budget but needs application-level quality checks; a smaller file is not proof of unchanged accuracy or faster end-to-end service.

Where the memory goes

For an illustrative attention configuration, KV bytes ≈ 2 × layers × KV heads × head dimension × bytes × tokens × batch. With 32 layers, 8 KV heads, dimension 128, two-byte values, 8,192 tokens and batch 8, the result is 8 GiB. This assumes the stated architecture and cache layout; it is not a universal model requirement. Admission control must account for growing sequences, cancellation and memory fragmentation, not merely successful model loading.

Keeping accelerators supplied with data

Separate compute, storage and management traffic according to workload and failure domains. Increasing GPU count can increase communication demands and leave expensive accelerators waiting for input. Measure prefill and decode separately, then test realistic request distributions, including long prompts.

Record p95/p99 latency, tokens per joule, saturation and recovery. Compare single-node and distributed serving with the same quality and latency target. The engineering decision is the smallest system meeting that target with operating headroom, not the largest advertised accelerator total.