GPU Networking Is the New Bottleneck

H100 to H200 was a 1.4x training throughput jump on most workloads. H200 to B200 lands closer to 2x. The trend per-card is real but it is decelerating relative to what the models actually need. The faster axis right now is interconnect. NVLink Gen 5 added 1.8 TB per second per GPU. InfiniBand NDR shipped 400 Gbps. Spectrum-X is at 800 Gbps. The compute layer is being limited by how fast each GPU can talk to every other GPU, not by how fast a single GPU can multiply matrices.

Why Interconnect Sets the Pace

Large-model training is dominated by all-reduce and all-to-all communication patterns. Every gradient sync between GPUs has to traverse the fabric. The proportion of training step time spent in communication has been climbing as models scale. For frontier models on multi-thousand-GPU clusters, communication overhead can reach 30 to 50 percent of total step time without proper fabric design.

That is the point at which the next GPU upgrade stops mattering. If the fabric is the bottleneck, faster compute on each card just means the cards spend more time waiting on each other.

NVIDIA acknowledged this directly with the GB200 NVL72 design. Inside a single NVL72 rack, all 72 GPUs share a unified NVLink domain, eliminating the network boundary entirely. The cost is that every NVL72 rack now requires roughly 120 kilowatts of power and direct-to-chip liquid cooling. The cost reflects what the fabric actually requires.

The Fabric Layers in 2026

The Provisioning Mistake

The mistake operators make is treating networking as the same line item it was in CPU-era data centers. A traditional enterprise rack might have terminated at 10 to 25 Gbps per server. AI training racks now need 400 to 800 Gbps per server, and a non-blocking fabric to back it up.

Non-blocking fabric means the cumulative bisection bandwidth between any two halves of the cluster is at least equal to the per-server bandwidth times half the cluster size. In practical terms, a 1,024-GPU cluster targeting 400 Gbps per GPU needs roughly 200 Tbps of bisection bandwidth across the fabric. That requires a substantial spine-leaf topology with high-radix switches at the spine.

Switch and Optic Math at 1,024 GPUs

A 1,024-GPU cluster with a 400 Gbps fabric typically needs 32 leaf switches (32 ports each toward GPUs, 32 toward spine) and 4 to 8 spine switches at high radix. Total switch count alone is 36 to 40 units, plus all the optics and fiber.

800 Gbps optics run roughly $1,500 to $3,500 per port at current pricing (LightCounting 800G Optics Market Report, 2025). A 1,024-GPU cluster needs 2,048 to 4,096 optic transceivers depending on fabric design. The optics line item alone often runs into seven figures.

Topology Choices That Compound

Beyond raw bandwidth, the topology itself decides what kinds of training workloads run efficiently. Two choices dominate at scale.

Fat Tree (Spine-Leaf)

Fat tree is the conventional topology. Every leaf switch connects to every spine switch with enough aggregate bandwidth that any-to-any traffic does not oversubscribe the fabric. Predictable performance, well-understood, but the spine switch cost scales aggressively with cluster size. At 4,000-plus GPUs, the spine layer alone can cost 5 to 8 million dollars in switches.

Dragonfly+ and Rail-Optimized

Some hyperscale clusters now run Dragonfly+ or rail-optimized topologies that exploit the structure of training communication patterns. By co-locating GPUs that frequently communicate (the same model parallelism group) on the same rail, the topology trades any-to-any bandwidth for tail latency on the patterns the workload actually uses. The result can be 30 to 40 percent lower fabric cost at the same effective training throughput.

The tradeoff is that the workload software has to be aware of the topology. Frameworks like NCCL handle this for standard training patterns, but anything outside the standard patterns can hit the rail boundary and stall.

What This Means for Site Selection

Two implications follow that change how operators evaluate facilities.

First, the cluster topology has to be designed before the racks are placed. Rack layout, fiber runs, and switch positioning are all downstream of fabric topology. A facility that locks in a hot-aisle layout before fabric design is making the choice for the operator.

Second, the per-rack power envelope is no longer the only constraint. A 120 kilowatt NVL72 rack also needs roughly 14 to 20 fiber drops at 800 Gbps each. Cable management, fiber tray sizing, and patch panel density all become first-class facility requirements alongside power and cooling.

The Implication for 2026

Operators evaluating colocation in 2026 should be asking about bisection bandwidth headroom and fiber density per cabinet just as carefully as power per rack. Facilities engineered for CPU-era enterprise traffic will not support multi-thousand-GPU training without significant rework. The retrofit cost on the fabric side runs in the same envelope as the cooling retrofit cost: 50 to 70 percent of new build per megawatt once you include cabling plant, switch room expansion, and downtime.

The fabric decision is no longer a downstream engineering choice. It is part of the deployment model from the first site visit forward.