Choose by model size
- Up to 13B parameters: one RTX PRO 6000 or one H100 handles most traffic, often with several replicas per card.
- 30–70B: one H200 (8-bit) or two H100s (16-bit). The RTX PRO 6000 serves 70B in 4-bit.
- 100B+ and mixture-of-experts: 8-GPU H200 or B200 nodes with tensor parallelism.
Software that gets the most out of the GPU
Use a serving engine with continuous batching and paged KV-cache, such as vLLM, SGLang or TensorRT-LLM. On the same hardware these can serve several times more requests than naive batching.
On demand or reserved
Steady production traffic is cheapest on a reserved cluster: 12-month terms are 10% below on-demand rates and 24–36 month terms 14% below. Keep on-demand GPUs for traffic peaks and experiments.





