The best GPU depends on the model, precision, software stack, and where it will run. Start with memory requirements and a benchmark from the workload you actually plan to use. Then compare complete systems at current prices.
The buying shortlist
| Product family | Consider it when | Check before ordering |
|---|---|---|
| NVIDIA H100 SXM | You are expanding a Hopper-based CUDA deployment | SXM server compatibility, available build, and support |
| NVIDIA H200 SXM | More GPU memory helps, and you want to remain on Hopper | Whether 141 GB per GPU is enough for the model and context |
| NVIDIA B200 SXM | You need more memory, FP4 capability, or newer interconnect | OEM chassis, cooling, software versions, and a workload benchmark |
| AMD MI300X | High memory capacity matters and ROCm fits your stack | Framework support and performance on your own workload |
| NVIDIA L40S | You need a PCIe GPU for graphics, video, or smaller inference jobs | Memory limit and server support |
| NVIDIA A100 | You are maintaining an existing A100 fleet | Current support, condition, warranty, and total cost |
NVIDIA lists 180 GB HBM3e per HGX B200 GPU, 141 GB for H200 SXM, and 80 GB for H100 SXM in its HGX reference architecture. The B200 can be part of an air- or liquid-cooled OEM server. Confirm the exact system.
Memory first, then speed
Model weights are only part of GPU memory use. Context length, batch size, attention cache, activations, and fine-tuning method also matter. Quantization may lower memory demand but can change quality or throughput. Build a memory estimate for your model before choosing a GPU count.
A spec-sheet FLOPS number will not predict tokens per second or training time by itself. Ask for results with the same model, precision, framework, batch size, and number of GPUs. Include networking when a job spans nodes.
Software and facility fit
CUDA-specific kernels and NVIDIA-only libraries can make an AMD migration costly. ROCm can be a good fit when your framework and model are supported. Validate your exact software stack before buying.
Server form factor matters. H100 PCIe, H100 SXM, and B200 SXM do not slot into the same platform. Ask the OEM for the server model, cooling method, rack power, NICs, and warranty. For B200, NVIDIA says HGX supports advanced air or liquid cooling.
Compare quotes
Hardware pricing and delivery need a current quote for the complete system. Compare current written quotes for complete systems, including the OEM part numbers, support, freight, and target delivery date. The cheapest GPU-only offer may be the wrong server build.
