- Ant Group’s Ling Team published a paper on June 18, 2026 introducing UFP4, a uniform 4-bit training recipe that replaces the E2M1 format used in NVIDIA Blackwell/Rubin and AMD MI350-series GPUs with uniform grids (E1M2 or INT4) to eliminate a systematic rounding error the researchers call Shrinkage Bias.
- Shrinkage Bias is caused by the geometric asymmetry of E2M1’s representable number bins: the resulting negative rounding error compounds across every layer of a network and worsens when the Random Hadamard Transform is applied, destabilising training in current FP4 methods.
- Across Dense 1.5B, MoE 7.9B and MoE 124B model runs, UFP4 consistently produced lower loss degradation relative to BF16 than E2M1 baselines, a result the team says argues directly for hardware vendors to add native E1M2/INT4 support in next-generation accelerators.
Ant Group‘s Ling Team has identified why FP4 training keeps breaking at scale, and the culprit turns out to be baked into the number format itself. Their paper, published June 18, 2026, traces chronic training instability in 4-bit pretraining to what they call Shrinkage Bias: a systematic rounding error that compounds across every layer of a network and gets worse, not better, when standard stabilisation techniques are applied. Their proposed fix, a training recipe called UFP4, swaps out the problematic format entirely.
FP4’s Promise and Persistent Pretraining Hurdles
Four-bit floating-point training is appealing for one straightforward reason: it cuts memory and compute costs well below what FP8, the current standard on NVIDIA H100/H200 GPUs, can offer. That gap matters enormously when you are pretraining models at scale, but the more aggressively you compress numerical precision, the more you expose the model to quantization error, overflow and underflow. With FP8 those risks are manageable. At FP4 they compound quickly.
The difficulty is not simply that FP4 represents fewer values, it is that the errors introduced by that limited range accumulate through hundreds of layers during a training run. Gradient signals, already noisy, degrade further with each pass. Several research groups have tried to paper over this with differentiable gradient estimators, outlier compensation and dynamic scaling, but the instability has persisted. The Ling Team paper argues those fixes have been treating symptoms rather than the underlying cause.
Unveiling the Geometric Roots of Shrinkage Bias
The root cause, according to Qian Zhao, Kunlong Chen and their colleagues, is geometric. The E2M1 format, the FP4 variant currently built into NVIDIA Blackwell/Rubin and AMD MI350-series hardware, distributes its representable numbers unevenly across the number line. That non-uniformity means the rounding that occurs when a value is quantized is not random noise. It is systematically biased toward smaller magnitudes, producing a consistent negative error the researchers call Shrinkage Bias.
Multiply a small bias by itself across dozens of matrix operations per layer, across hundreds of layers, and the effect is no longer small. The paper shows the accumulation is multiplicative, not additive, which explains why instability tends to appear gradually before suddenly becoming acute. The picture gets worse when you factor in the Random Hadamard Transform (RHT). RHT is widely used in narrow-precision training to spread outlier values across dimensions, making them easier to quantize cleanly. The Ling Team shows that in E2M1, RHT interacts with the format’s geometric asymmetry to amplify Shrinkage Bias rather than counteract it. A tool imported to improve stability is instead making things worse.
Systemic Instability: How Shrinkage Bias Disrupts Training
The practical consequence is that gradient propagation breaks down. The three core matrix multiplications in transformer training, involving weights, activations and output gradients, each accumulate Shrinkage Bias independently. By the time those signals combine in a backward pass, the numerical fidelity needed for stable convergence is gone. The paper also points to a secondary problem: dominant activation outliers tend to cluster around a coherent mean bias, which inflates the dynamic range a quantizer must cover and compresses everything else into the narrowest bins. That further degrades quantization quality for the majority of values, which are not outliers at all.
Prior FP4 training efforts addressed pieces of this, outlier compensation here, a modified estimator there, but none diagnosed the geometric origin of the bias. That gap is what the Ling Team paper closes.
The UFP4 Recipe: A Uniform Approach to FP4 Stability
UFP4’s central move is to abandon E2M1 entirely. Instead, it uses uniform grids, E1M2 or INT4, where representable values are evenly spaced. Even spacing means no geometric asymmetry, which means no Shrinkage Bias by construction. That single substitution removes the root cause rather than compensating for it.
The recipe then applies RHT to all three training matrix multiplications. On a uniform grid, RHT works as intended: it spreads outlier energy and improves how well the quantizer bins represent the actual value distribution. The final piece is stochastic rounding, the technique of randomly rounding up or down to prevent gradients from collapsing, applied selectively to only the output gradient component (dY) rather than across all three operations. That targeted application preserves the variance that makes stochastic rounding useful, without reintroducing the instability it can cause when applied more broadly. Together, the three design choices are mutually reinforcing: uniform grid, full RHT coverage, restricted stochastic rounding.
Empirical Validation and Hardware Implications
The team tested UFP4 across three model classes: a Dense 1.5B, an MoE 7.9B and an MoE 124B. Across all three, UFP4 produced lower loss degradation relative to BF16 than the E2M1 baselines in long-run pretraining. The results held up through scaling-law analyses and ablation studies designed to isolate the contribution of each component. The 124B MoE result is the most consequential, it is the scale at which training instability in FP4 has historically been most acute, and where the cost of a failed run is highest.
The hardware implication the paper draws is direct: if uniform 4-bit grids perform better and the instability of E2M1 is geometric rather than incidental, then next-generation accelerators should support E1M2 and INT4 as first-class training primitives alongside E2M1, not as afterthoughts. That is a concrete ask directed at AMD and NVIDIA’s roadmap teams, not a general observation about the direction of AI hardware. Whether either company moves on it depends on yield, die area and customer demand, but the paper gives hardware architects a specific technical basis to evaluate. For teams planning pretraining runs on Blackwell or MI350-class systems, the finding is worth weighing now, since compute decisions made at the hardware selection stage constrain what training formats are even available. For more coverage of AI research and breakthroughs, visit our AI Research section.
