Start with a measured workload. Model parameters alone do not tell you how many GPUs, how much network, or how much storage a training job needs.
Define the job
Record the model architecture, training method, dataset size, sequence length, precision, target completion time, checkpoint frequency, and software stack. Estimate memory for weights, optimizer states, gradients, activations, and temporary buffers. Fine-tuning and full pretraining have different requirements.
Choose a node
Select a validated OEM server with enough GPU memory for the planned parallelism. Check CPU, system RAM, local storage, GPU interconnect, and software support. NVIDIA's HGX reference architecture lists 80 GB per H100 SXM, 141 GB per H200 SXM, and 180 GB per B200 SXM. An eight-GPU B200 node has 1.44 TB of GPU memory.
For multi-node jobs, network topology matters. Ask the OEM or integrator how many NICs and what fabric are required for the training framework and target node count. Avoid selecting network speed from a generic price-per-port table.
Measure scaling
- Run a single-node benchmark on the target model and precision.
- Measure GPU utilization, step time, memory use, and checkpoint I/O.
- Test a two-node configuration with the planned network.
- Estimate the node count needed for the target completion time.
- Leave capacity for failures, validation runs, and data preparation.
The scaling curve often changes as nodes are added. A straight-line estimate from one GPU to hundreds of GPUs is unreliable.
Facility and procurement
Ask for the exact server model, maximum power, cooling method, rack requirements, warranty, and support. NVIDIA says HGX B200 can pair with air or liquid cooling; the OEM server decides the installation requirement.
Budget from current written quotes for complete servers, networking, storage, delivery, and facility work. Cluster pricing and lead time depend on the final build and available allocation.
