Cloud vs Iron: Where to Run Your SOTA Models in 2026

Your team just shipped an AI feature. Users love it. Traffic is climbing. The OpenAI bill is eating your runway. Someone asks the question that every AI team eventually faces: “Should we run this ourselves?”

This is the guide that answers it --- not with theory, but with real numbers, real hardware, and a decision framework you can use today.

TL;DR

• Cloud wins for most teams. If your GPU utilization sits below 70% (and it probably does), renting is cheaper and simpler.
• Specialist GPU clouds are 40—85% cheaper than AWS/GCP/Azure. Use CoreWeave, Lambda Labs, RunPod, or Spheron instead of hyperscalers unless you need specific compliance features.
• Start with vLLM. It has the widest model support, no compilation step, and competitive throughput. Move to TensorRT-LLM only when your model is stable and you need maximum performance.
• H100 SXM5 is the practical default GPU in 2026. The B200 is faster but supply-constrained.
• On-premise breaks even at 80%+ sustained utilization over a 3-year horizon --- or in under 4 months for high-utilization workloads.

The Scenario

Imagine a ten-person AI startup in mid-2026. They have been running Qwen3.5-35B behind a chatbot API, paying per token to a model-as-a-service provider. Their monthly inference bill just crossed $8,000. The CFO wants to cut costs. The CTO wants to own the infrastructure. The engineering team wants to stop worrying about rate limits and data leaving their VPC.

They need to decide: rent GPUs from a cloud provider, or buy their own hardware?

This is the decision we will walk through. Along the way, we will cover which GPU to pick, which inference engine to use, which cloud provider offers the best deal, and which open-weight models are actually worth self-hosting right now.

The Utilization Rule

Before we compare prices or hardware, there is one number that matters more than anything else: your GPU utilization rate.

The research is clear:

Most production teams operate at 40—65% utilization due to traffic variability. This means cloud wins for the majority of use cases. But if you are running batch inference jobs 24/7 or serving a high-traffic API with predictable load, owning hardware becomes compelling.

The break-even math is surprisingly aggressive for high-utilization workloads: on-premise infrastructure can break even in under 4 months compared to hyperscaler pricing, according to Lenovo’s 2026 TCO whitepaper.

Sources: Spheron — On-Premise vs Cloud, April 2026 | Lenovo Press — TCO Whitepaper, 2026

Which GPU in 2026?

The GPU market in 2026 is dominated by NVIDIA’s Hopper and Blackwell architectures. Here is the landscape:

Practical recommendations:

• Default choice: H100 SXM5. Best availability, proven in production, supported by every inference engine.
• Future-proofing: B200, if you can get one. It delivers roughly 4x the inference throughput of an H100, but supply remains constrained through mid-2026.
• Budget option: A100 spot instances. Performance is solid and prices are falling below $1/hr on some providers.
• Local dev: RTX 4090. Incredible FLOPS-per-dollar for models under 24 GB, but not suitable for production serving.

The buy vs rent math for one H100: Buying an H100 costs approximately $30,000. Renting one at $5/hr means break-even at roughly 6,000 hours --- about 3.4 years at 5 hours per day --- before accounting for power, cooling, and facilities. This is why utilization matters so much.

Sources: Inworld AI — GPU Cloud Comparison, April 2026 | WhiteFiber — GPU Guide

Cloud Providers: Who Has the Best GPU Pricing?

This is where the market gets interesting. Specialist GPU cloud providers are 40—85% cheaper than the hyperscalers for equivalent GPU workloads. AWS, GCP, and Azure add egress fees, storage premiums, and networking charges that inflate the real cost significantly.

Pricing Snapshot (May—June 2026)

The takeaway: If you are paying AWS or GCP prices for GPU inference, you are overpaying. Switch to a specialist provider and your costs drop by half or more. Use hyperscalers only when you have specific compliance requirements (HIPAA, FedRAMP) or deep integration with their ecosystem (GKE, SageMaker).

Sources: Spheron — GPU Pricing, May 2026 | Inworld AI — GPU Cloud Comparison, April 2026

The Inference Stack: vLLM vs SGLang vs TensorRT-LLM

Choosing the right GPU is only half the battle. The software stack that serves your model determines your actual throughput, latency, and operational complexity. In 2026, the inference engine landscape has consolidated around three serious contenders.

The 2026 Recommendation Hierarchy

1. Start with vLLM --- widest model support, no compilation step, competitive throughput, best documentation. This is the safe default for any team.
2. Move to TensorRT-LLM --- when your model is stable for months and you need maximum throughput. Expect a 20—30% throughput gain over vLLM, but a 28-minute compilation step that must be rerun whenever the model changes.
3. Use SGLang --- when individual request latency matters most (real-time chat, structured output generation, shared-prefix workloads). It excels at minimizing time-to-first-token.
4. Avoid TGI (HuggingFace) --- officially in maintenance mode. HuggingFace themselves recommend migrating to vLLM or SGLang.

Benchmark Comparison

Benchmark note: On H100 SXM5, TensorRT-LLM delivers the best throughput and latency but requires compilation. On the Blackwell B200, TensorRT-LLM reaches 1,000 tokens per second per user on Llama 4 Maverick --- a staggering number that makes it the clear choice for high-volume production workloads on next-gen hardware.

Sources: Spheron — Inference Benchmarks, March 2026 | The AI Engineer — Inference Showdown, March 2026

Models Worth Self-Hosting in 2026

Not every model justifies the infrastructure investment. These are the open-weight models that make self-hosting worthwhile in 2026:

Why these models specifically? They are Mixture-of-Experts architectures, which means they activate only a fraction of their total parameters during inference. A 35B MoE model with 3B active parameters runs nearly as fast as a dense 3B model while delivering quality closer to a 30B+ dense model. This makes them incredibly efficient to self-host --- you get frontier-tier quality at a fraction of the compute cost.

The Nemotron-30B Omni deserves special mention: its Mamba2 hybrid architecture and 1M context window make it uniquely suited for multimodal workloads (text, image, audio) on a single GPU.

The Decision Matrix

Here is the consolidated decision framework based on your specific scenario:

Back to Our Startup

Let us return to the scenario we opened with. That ten-person startup with the $8,000/month inference bill?

Here is what they would do:

1. Switch from MaaS to self-hosted inference. Rent an H100 SXM5 on Spheron at $1.03/hr spot (or $2.90 on-demand). Run Qwen3.5-35B-A3B through vLLM. Their per-token cost drops from the API’s premium pricing to roughly $0.47 per million tokens.

2. Use a specialist GPU cloud, not a hyperscaler. Spheron, RunPod, or Lambda Labs will save them 40—85% compared to AWS or GCP for the same GPU.

3. Start with vLLM, no optimization needed yet. At their traffic level, vLLM’s throughput on a single H100 is more than sufficient. They can migrate to TensorRT-LLM later if throughput becomes a bottleneck.

4. Monitor utilization. If they consistently hit 80%+ utilization, they should start planning an on-premise purchase. If they sit at 40—60%, staying on cloud spot instances is the right call.

The result: their monthly inference bill drops from $8,000 to roughly $800—1,500, depending on traffic patterns. That is a 5—10x cost reduction with better latency, full data control, and no rate limits.

Why This Stack?

A few words on the reasoning behind these recommendations, in case you are weighing alternatives:

Why specialist clouds over hyperscalers? AWS, GCP, and Azure charge premium prices for GPU instances because they bundle networking, storage, compliance, and ecosystem lock-in into the price. If you do not need those extras, you are subsidizing features you do not use. Specialist providers give you the same GPU at a fraction of the cost.

Why vLLM as the default? It works with virtually every model architecture out of the box. No compilation step means you can swap models in minutes. The documentation is excellent, the community is active, and the performance is competitive. You only need TensorRT-LLM’s 20—30% throughput gain when you are pushing the limits of your hardware --- and when you are, the 28-minute compilation step is worth it.

Why MoE models? Mixture-of-Experts architectures give you the quality of a large model at the inference cost of a small one. Qwen3.5-35B-A3B activates only 3 billion of its 35 billion parameters per token. This means you get near-frontier performance on hardware that would struggle with a dense 35B model.

Why not just use Ollama for everything? Ollama is fantastic for local development and experimentation. It is not designed for production serving at scale. It lacks the batching, scheduling, and throughput optimizations that vLLM, SGLang, and TensorRT-LLM provide. Use Ollama on your laptop; use vLLM in production.

Sources

The GPU market moves fast. The inference stack evolves faster. But the fundamental tradeoff --- rent vs own, cloud vs metal, flexibility vs control --- is the same decision every AI team faces as they scale. Pick the right tool for your utilization, your timeline, and your constraints. Everything else follows.