- Global LLM inference spending is projected to reach $255 billion by 2030, surpassing training budgets.
- Reasoning-heavy agentic workloads can push effective token costs 50 to 100 times higher than per-token API prices suggest, because each user query can trigger thousands of internal reasoning tokens before any output is produced.
- Self-hosting a four-GPU H100 server can exceed $400,000 upfront, and hidden costs, power, egress fees and engineering headcount, often determine whether that bet pays off.
Global LLM inference spend is on track to hit $255 billion by 2030, overtaking training budgets by a wide margin. That shift is already visible in enterprise billing: some models now deliver GPT-4-level output for around $0.40 per million tokens, yet monthly API bills for large organisations are still climbing into the tens of millions.
The Thinking Token Problem
Modern reasoning models work through problems step by step before producing any visible output. A single user query can trigger 10,000 internal reasoning tokens to generate a 200-token answer, a 50x multiplier on actual token spend, sometimes reaching 100x in complex agentic chains. Gartner’s March 2026 analysis found that agentic models require 5 to 30 times more tokens per task than standard chatbots. Google’s Gemini 3.1 Flash-Lite, launched March 3, 2026, lists at $0.25 per million input tokens and $1.50 per million output tokens, competitive on paper, but if a reasoning task generates 50x the expected output tokens internally, the effective cost per visible result scales accordingly. The sticker price is real; it just does not describe what you actually spend.
The Hardware Stack
Nvidia‘s Hopper H100 GPUs remain the workhorse in most data centres, with cloud rental rates running roughly $2.85 to $3.50 per GPU-hour on budget-tier providers and $5 to $8 on hyperscalers as of early 2026. New units cost $25,000 to $40,000 depending on variant. The H100’s Transformer Engine accelerates large language models by up to 30x over prior-generation hardware, according to Nvidia’s specifications.
The B200 GPU and GB200 NVL72 systems deliver cheaper inference than Hopper. Blackwell-backed infrastructure puts models like GPT-OSS-120B at a lower cost per million tokens, though the actual gap depends on workload and provider. For teams weighing inference cost against compute architecture, the shift toward inference-time compute scaling is worth understanding before committing to hardware.
Cloud vs. Self-Hosting
Managed API providers, OpenAI Anthropic Google and open-source hosts like Together AI, Fireworks AI and DeepInfra, charge on consumption, typically per million tokens, with no upfront capital commitment. As of May 2026, DeepInfra lists Llama 3.3 70B at $0.23 per million input tokens. The convenience premium is real, but for high-volume workloads the economics eventually invert.
Self-hosting demands significant upfront spend: a four-GPU H100 server can exceed $400,000 to purchase outright. For sustained high-volume workloads on comparable open-source models, self-hosting can become more cost-effective than API pricing, according to operational estimates, though the actual gap varies significantly by workload and provider.
The hidden costs are where self-hosting calculations fall apart. An H100 SXM5 draws up to 700W at peak and around 100W at idle; power, cooling and rack space compound fast. Egress fees add further pressure: AWS and Azure charge $0.087 to $0.09 per GB for outbound data, GCP charges $0.11 to $0.12 per GB. A deployment generating 1 TB of inference output per day can spend $2,600 to $3,600 per month on data transfer alone, before touching compute. GPU failure rates of 5 to 10% annually add replacement cost and downtime risk. The build-versus-buy decision is as much about engineering headcount as it is about hardware price.
API Price Competition
Since March 2023, the cost of comparable API performance has dropped by more than 90%. Google’s Gemini 3.1 Flash costs $0.10 per million input tokens and $0.40 per million output tokens as of April 2026, a 99.7% reduction from GPT-4’s launch price. That compression is driven by OpenAI, Anthropic, Google, xAI and Meta competing on price alongside dozens of open-source model hosts. Specialised GPU clouds, GMI Cloud, Lambda Labs, RunPod, CoreWeave, quote direct per-GPU rates that can run well below hyperscaler bundles.
Custom silicon is starting to change the latency equation. Groq‘s Language Processing Units (LPUs) deliver inference speeds the company claims run around 10x faster than H100s for Llama-family models, an advantage that matters most for latency-sensitive applications like voice agents. The practical implication: production LLM deployments need active workload routing across a portfolio of providers and hardware types rather than a single committed stack. For a closer look at how alternative silicon architectures are reshaping inference the Cerebras-AMD partnership is instructive.
Optimisation Levers
Running Llama 3 70B at INT4 on a single A100 80GB GPU cuts per-token costs compared with running FP16 across multiple A100s. The efficiency gain comes from fitting the model on fewer, more efficiently utilised chips. Blackwell’s native FP8 and NVFP4 support pushes this further, and quantisation techniques at FP8 and INT4 can bring smaller models to performance comparable to much larger ones on standard NLP tasks.
Switching a production workload from a top-tier proprietary model to an equivalent open-weight model via a provider like Inference.net can reduce monthly API spend substantially. How much depends heavily on workload composition and quality requirements, teams should benchmark their own traffic before assuming headline savings apply. For teams looking to profile actual inference spend before switching stacks, nvidia-smi and agentic workflow profiling can surface where costs are actually accumulating.
The MLOps Bill Nobody Budgets For
Maintaining a production LLM stack requires a significant share of a senior infrastructure engineer’s time. At a fully loaded annual cost of $180,000 to $250,000 for that role, LLM infrastructure carries a substantial labour overhead before touching the application layer.
That time covers GPU driver updates, firmware patches, hardware troubleshooting, model versioning, monitoring setup (Prometheus and Grafana for GPU metrics, latency and throughput) and compliance work. Even on managed API services, monitoring, prompt engineering and continuous model evaluation require dedicated headcount. Routing across providers, maintaining caching layers and running quality evaluations in production adds further demand. API price cuts save money on the compute line; they do not reduce the engineering burden of running production AI reliably.
Purpose-Built Silicon
Groq’s LPUs deliver hundreds of tokens per second for models like Llama 3.1 8B at speeds that outpace H100s for certain inference patterns. The trade-off is specificity: LPUs excel at particular throughput profiles rather than the broad flexibility of GPU deployments, which limits their appeal outside latency-critical use cases like real-time voice.
Smaller models are also closing the capability gap. Qwen2.5-VL-7B-Instruct, GLM-4-9B-0414 and Meta’s Llama 3.1-8B-Instruct can deliver enterprise-grade performance on significantly less hardware, a single Nvidia L4 GPU or, at the lower end, consumer-grade cards like the RTX 5060 Ti 16GB, a Blackwell-generation card with 16GB of VRAM suited to running mid-sized models locally. Procurement decisions are moving away from “most powerful available” toward “most efficient for this specific workload.” GPU, FPGA and ASIC combinations let teams tailor hardware to inference patterns rather than over-provisioning for peak load. Electricity costs compound over years of continuous operation, and even modest per-watt improvements at data centre scale translate to material savings.
