GPU Cloud at $2.25/hr or $12.29/hr: The Infrastructure Layer That Determines Which Price You Pay
The same H100 GPU rents for $1.38/hr on one platform and $12.29/hr on another. That's not a typo and it's not a different product. It's the same 80GB HBM3 chip running the same CUDA cores. The difference is everything underneath: the facility, the power contract, the cooling, and the operating model.
Mordor Intelligence valued the GPU-as-a-Service market at $7.34B in 2026, projecting growth to $25.94B by 2031 at a 28.7% CAGR. That's a market nearly quadrupling in five years. But the more interesting number isn't the total. It's the spread. Thunder Compute's H100 pricing index shows current market rates ranging from $1.38/hr to $12.29/hr for the same GPU. A 9x spread on identical silicon tells you that the chip isn't what you're really paying for.
You're paying for infrastructure. And infrastructure costs vary wildly depending on where it sits, how it was built, and who operates it.
Deconstructing the GPU Cost Stack
Every GPU hour you purchase is a blend of three cost layers. The proportions shift depending on the provider's infrastructure model, but the categories are consistent across every operator in the market.
GPU-Hour Cost Stack Breakdown
- Silicon (GPU Hardware + Depreciation): 35-45%
- Facility + Power (Build Cost + Electricity): 40-50%
- Operations (Staff, Software, Networking, Margin): 10-15%
- Infrastructure-Dependent Cost (Facility + Power + Ops): 55-65% of total
Silicon costs are roughly equal across providers. NVIDIA's pricing for H100 SXM modules doesn't vary much between buyers at scale (list price around $25,000-$30,000 per GPU). The B200, NVIDIA's next-generation GPU, carries an estimated list price of $30,000-$40,000. These numbers are relatively fixed. What isn't fixed is everything else.
The 55-65% of cost that's infrastructure-dependent is where the pricing spread originates. And that's where infrastructure decisions made years before a GPU is ever powered on determine whether you're charging $2.25/hr or $12.29/hr.
Power: The Variable That Matters Most
An H100 GPU running at full utilization draws approximately 700W. Add the server overhead (CPU, memory, networking, storage) and you're at roughly 1kW per GPU. Run that GPU 24/7 and you're consuming 8,760 kWh per year (per GPU). Scale that to a 1,000-GPU cluster and you're looking at 8.76 million kWh annually just for the IT load.
But the IT load isn't the total power draw. PUE (Power Usage Effectiveness) multiplies it. A facility running at PUE 1.58 (the industry average per the Uptime Institute) consumes 58% more power than the IT equipment itself, with that overhead going to cooling, lighting, power conversion losses, and facility systems.
| PUE | Overhead | Total Power (1,000 GPUs) | Annual Cost @ $0.05/kWh |
|---|---|---|---|
| 1.58 (Industry Average) | +58% | 13.84M kWh | $692,040 |
| 1.30 (Modern Air-Cooled) | +30% | 11.39M kWh | $569,400 |
| 1.10 (Optimized Liquid-Cooled) | +10% | 9.64M kWh | $481,800 |
That's a $210,240 annual difference between PUE 1.58 and PUE 1.10, on just 1,000 GPUs at $0.05/kWh. Scale to 10,000 GPUs and you're talking $2.1M per year. Over a 5-year GPU depreciation cycle, that's $10.5M in power savings alone from better facility design.
But the rate matters just as much as the PUE. Power costs vary by more than 3x across U.S. markets. TVA (Tennessee Valley Authority) territory offers industrial rates around $0.04-0.06/kWh. Northern Virginia, where 70% of the world's internet traffic passes through, sits at $0.07-0.10/kWh. California markets can reach $0.15-0.20/kWh for commercial power.
Annual Power Cost: 1,000 H100 GPUs by Market and PUE
- -- TVA Territory @ $0.05/kWh, PUE 1.10: $481,800/year. That's $0.055 per GPU-hour in power cost alone.
- -- Northern Virginia @ $0.08/kWh, PUE 1.30: $911,520/year. Power cost per GPU-hour jumps to $0.104.
- -- California @ $0.17/kWh, PUE 1.58: $2,352,480/year. Power alone costs $0.269 per GPU-hour, nearly 5x the TVA scenario.
When power cost alone varies by 5x based on location and facility efficiency, the downstream impact on GPU pricing is enormous. That's before you factor in facility build cost.
Facility Cost: $4-8M/MW Modular vs. $11.3M/MW Traditional
Turner and Townsend's 2025 data center cost index places the average build cost at $11.3M per MW of IT capacity. That average includes traditional stick-built facilities with custom engineering, long construction timelines, and the current supply chain premiums on electrical equipment (transformer lead times alone have hit 128 weeks, as we covered in a recent analysis).
Pre-engineered modular approaches reduce that figure significantly. Modular market costs cluster in the $4-8M/MW range (with T&T's 2025 benchmark around $4.98M/MW), varying with cooling architecture, rack density, and site integration scope. Factory-direct, cold-plate DLC platforms sit at the low end of that range - a 55-65% reduction compared to the traditional approach.
Facility Cost Impact on GPU Pricing (10MW Facility, 5-Year Depreciation)
Traditional Build (per MW)
$113M total for 10MW facility. Amortized over 5 years at 10,000 GPUs: $2.58/GPU-hour in facility cost alone. That's before power, before staff, before the GPU itself.
Pre-Engineered Modular (per MW)
$49.8M total for 10MW facility. Same amortization: $1.14/GPU-hour in facility cost. That's $1.44/GPU-hour cheaper before you've touched any other variable.
A $1.44/GPU-hour facility cost advantage doesn't sound dramatic in isolation. But multiply it across 10,000 GPUs running 24/7 for a year and it's $126.1M in cost difference. That margin is what separates a profitable GPU cloud operator from one that's subsidizing compute with venture capital.
The Three Infrastructure Models: Cloud vs. Colo vs. Build-Own
Deloitte's Tech Trends 2026 report found that 85%+ of enterprises now operate in multi-cloud environments. But "multi-cloud" doesn't mean every workload belongs on every platform. GPU-intensive workloads, in particular, are gravitating toward infrastructure models that optimize for the cost stack variables we've just broken down.
| Factor | Hyperscale Cloud | GPU Colocation | Build-Own |
|---|---|---|---|
| H100 Hourly Rate | $8.00-$12.29/hr | $2.25-$4.50/hr | $1.38-$2.50/hr* |
| Upfront Capital | $0 (pay-as-you-go) | GPU hardware + setup | Full facility + hardware |
| Time to Deploy | Minutes | 2-8 weeks | 26-128 weeks |
| Minimum Commitment | None to 1-3 years | 12-36 months | 5+ years (asset life) |
| Operational Burden | None (managed) | Hardware management | Full stack (facility + hardware) |
| 3-Year TCO (1,000 H100s) | $70M-$107.7M | $19.7M-$39.4M | $12.1M-$21.9M* |
*Build-own rates assume amortized facility cost, excludes capex recovery. Sources: Thunder Compute H100 Index, vendor pricing (Q1 2026)
The numbers tell a clear story, but they don't tell the whole story. Hyperscale cloud's premium buys you instant availability, managed operations, and zero capital commitment. For experimental workloads or burst demand, that premium is worth it. Nobody should build a data center to run a two-week fine-tuning experiment.
But for sustained GPU workloads (production inference serving, ongoing training runs, continuous model development) the math shifts dramatically. At 70%+ utilization over 12+ months, colocation starts saving 50-70% compared to hyperscale cloud. And build-own, for operators with the capital and expertise, offers 70-85% savings at scale.
The B200 Generation: Higher Power, Higher Stakes
NVIDIA's B200 GPU pushes TDP to 1,000W, a 43% increase over the H100's 700W. Neo-cloud providers have started listing B200 capacity from $2.25/hr, but the infrastructure requirements are materially different from H100 deployments.
B200 Infrastructure Impact vs. H100
Power Per GPU: 700W to 1,000W (+43%)
An 8-GPU B200 server draws approximately 10-11kW, up from 7-8kW for H100. Per-rack density at 42U pushes toward 100kW+, requiring liquid cooling as a baseline rather than an option.
Cooling: Liquid Required, Not Optional
At 1,000W per GPU, air cooling is not viable for production-density racks. Direct-to-chip liquid cooling becomes the minimum standard. This means facilities built for H100 air-cooled deployments may not support B200 without retrofit.
Electrical: 480V Standard, Larger Feeders
Higher per-rack power draws require larger electrical feeders, higher-capacity PDUs, and more switchgear capacity. Facilities designed for 20-30kW/rack can't simply swap in B200 hardware.
The B200 generation amplifies the infrastructure cost advantage. At 43% higher power draw per GPU, the gap between a facility running PUE 1.10 and one running PUE 1.58 widens proportionally. Poor infrastructure efficiency doesn't just cost more. It costs more on every GPU, every hour, compounding over the asset's life.
Where the Market Is Heading
The GPU cloud market's growth from $7.34B to $25.94B by 2031 will be driven by operators who control their infrastructure costs, not by those who resell hyperscale compute at a markup. The pricing spread we're seeing today ($1.38 to $12.29 for the same GPU) is a direct reflection of infrastructure efficiency differences.
Three trends are shaping which operators will capture that growth:
1. Power Location Arbitrage
Operators are moving toward low-cost power markets: TVA territory, Pacific Northwest hydro, Texas wind corridors. The 3x+ power cost difference between California and Tennessee translates directly into per-GPU-hour pricing advantage. This isn't new for data centers broadly, but it's accelerating for GPU-specific facilities because power is a larger share of GPU operating cost than it is for general-purpose compute.
2. Purpose-Built Facilities Over Retrofit
Retrofitting a 10kW/rack facility for 40-100kW/rack GPU density is technically possible but economically questionable. The cost of adding liquid cooling, upgrading electrical distribution, and reinforcing structural capacity for heavier racks often exceeds 60% of building new. Operators building purpose-designed GPU facilities from the start are achieving PUE 1.10-1.20 at a fraction of the retrofit cost.
3. Modular Construction for Speed and Cost
At $4-8M/MW for the pre-engineered modular market (T&T 2025 benchmark: ~$4.98M/MW) vs. $11.3M/MW for traditional build, the capital efficiency gap is too wide to ignore. The timeline advantage (26-40 weeks vs. 128 weeks for traditional electrical procurement) means modular operators can deploy capacity 2-3x faster. In a market growing at 28.7% CAGR, speed to market is a revenue multiplier.
The operators who'll dominate GPU cloud pricing over the next five years aren't necessarily the ones with the most GPUs. They're the ones with the lowest infrastructure cost per GPU-hour. And that cost is determined by decisions made at the facility level: where to build, how to build, what power rate to secure, and what PUE to engineer for.
Silicon is a commodity input. Infrastructure is the moat.
References
Additional data from: U.S. Energy Information Administration (power rates by state), TVA industrial rate schedules, vendor pricing sheets (Q1 2026).
Calculate Your GPU Infrastructure TCO
Compare cloud, colocation, and build-own costs for your specific GPU workload. Factor in power rates, PUE, facility cost, and utilization to find your optimal infrastructure model.