- AMD’s Instinct roadmap runs MI325X (Q4 2024), MI350 (2025) and MI400 (2026), setting up an annual release cycle aimed directly at Nvidia’s Hopper and Blackwell generations.
- The MI300X’s 192GB of HBM3 fits models onto a single GPU that would otherwise need two H100s, cutting interconnect complexity in large-model serving deployments.
- CUDA holds a 10-30% performance lead in general AI training, but ROCm has reached near parity for LLM inference via vLLM and SGLang, with Meta’s production adoption of AMD GPUs as the clearest proof point.
AMD’s MI300X carries nearly 2.5 times the memory of an H100, and that gap is doing real work in production. At Computex 2024, AMD detailed an annual Instinct cadence stretching to 2026, each generation aimed at Nvidia’s latest silicon. The software gap with CUDA is real, but it is closing faster than most expected.
MI300X vs. H100
AMD‘s Instinct MI300X, launched in late 2023 carries 192GB of HBM3 memory and 5.3 TB/s of bandwidth. That dwarfs the H100’s 80GB HBM3 at 3.35 TB/s and beats even the H200’s 141GB HBM3e on raw capacity. For large language models with long context windows, that headroom is practical: a model that would otherwise span two H100s fits on a single MI300X, cutting interconnect complexity and simplifying the serving architecture.
Benchmarks from Chips and Cheese in June 2024 showed the MI300X outperforming the PCIe H100 on cache bandwidth across its four-level hierarchy: L1, scalar, L2 and a 256MB Infinity Cache acting as L3. AMD claims a 20% inference advantage over the H100 on Llama 2 13B in single-GPU setups, rising to 60% in an eight-GPU cluster, according to the company’s own testing. Training is a different story. SemiAnalysis testing in December 2024 found the MI300X, even with custom software optimisations and direct AMD engineering support, reached about 75% of H100/H200 training throughput, approximately 620 TFLOP/s in BF16 against the H100/H200’s roughly 720 TFLOP/s, a 14% deficit despite higher advertised peak figures.
Taking on Blackwell
The MI350, built on CDNA 4 and launched in 2025 is AMD’s answer to Nvidia‘s Blackwell generation.
Blackwell is not standing still. The B200, a single-die design running at 1,000W TDP, offers 192GB of HBM3e with up to 8 TB/s bandwidth and fifth-generation NVLink at 1.8 TB/s per GPU, double the H100/H200’s interconnect. Nvidia claims up to 4x faster training and 15x faster inference versus the H100, according to the company. AMD’s MI400 series, scheduled for 2026, is said to target up to 10x the performance of the MI355X, but that figure comes from AMD’s own roadmap projections and has not been independently verified.
The ROCm Gap
CUDA launched in 2007 and has had nearly two decades to accumulate optimised libraries, tooling and developer familiarity. cuDNN, cuBLAS, FlashAttention 3 (Hopper-specific), TensorRT-LLM and NVIDIA NIM containers are all deeply integrated with the platform. SemiAnalysis has noted that depth does not disappear quickly, a challenge that affects all non-Nvidia accelerators competing in this space.
ROCm has closed more ground in the past 18 months than in the several years before. PyTorch officially supports ROCm, and its compatibility layer lets most CUDA-based PyTorch models run on AMD hardware with minimal code changes. For LLM inference specifically, using vLLM or SGLang, ROCm has reached near parity with CUDA. Meta’s adoption of AMD GPUs for its ML infrastructure is the clearest signal that ROCm is stable enough for large-scale production. The gap that remains is concentrated in training: 2025 benchmarks put CUDA 10-30% ahead of ROCm in general AI workloads, a difference attributed to software maturity rather than underlying hardware. For deep learning training in particular, CUDA’s cuDNN integration and Tensor Core optimisations still carry a 20-30% advantage over ROCm’s rocBLAS/MIOpen stack on comparable hardware.
The Cost Trade-off
Cost is where AMD makes its clearest case. The MI300X’s memory density means fewer GPUs per deployment for large models, which lowers unit count, reduces interconnect hardware and simplifies rack layout. For memory-bound workloads, long-context LLMs chief among them, the cost per unit of memory bandwidth favours AMD. Teams weighing LLM inference costs will find that advantage most pronounced at the point where model size forces a multi-GPU split on H100 hardware.
Nvidia claims the B200’s higher memory bandwidth of up to 8 TB/s and fifth-generation NVLink will deliver greater throughput in high-concurrency serving scenarios, according to the company’s Blackwell launch materials. Whether that translates to a lower total cost per query at scale depends heavily on workload and pricing, neither of which Nvidia has disclosed in comparable terms.
