Two papers published on Hugging Face Daily Papers (July 22-23, 2026) address the same bottleneck from different angles. SLAI T-Rex documents the full post-training of DeepSeek-V4, a trillion-parameter MoE, on an Ascend NPU SuperPOD cluster: severe memory pressure, non-overlapped communication, inefficient kernels. “Where Should Optimizer State Live?” quantifies the root problem: in a 6.78 billion parameter MoE, AdamW accumulates 50.6 GB of optimizer state to update 12.6 GB of weights. Optimizer state is the largest item in the memory budget.

Why it matters to you: as we reported on July 22, AMD committed 5 billion and 2 GW of GPUs to Anthropic. This is the technical side of that investment. Training models at trillion scale requires more than buying chips: the memory needed by the training algorithm to track gradients grows faster than the parameters themselves. SkewAdam, the optimizer from the second paper, assigns different precisions to the three components of a MoE (dense backbone, experts, router) based on their gradient statistics. It cuts 97.4% of optimizer state: from 50.6 GB to 1.29 GB. Peak memory drops from 81.4 GB to 31.3 GB, bringing billion-parameter MoE training within the budget of a 40 GB GPU. For those building models or evaluating infrastructure constraints, this is the detail that separates a cluster that runs from one that stays idle.

In detail

Optimizer state explains why training large models is expensive. When you train a model with AdamW (the standard optimizer for LLMs), the algorithm doesn’t store just weights: it also keeps a copy of past gradients (first moment) and an estimate of their variance (second moment). For each parameter you need two additional values in memory, plus the parameter itself. On a dense model this doubles or triples needed memory. On a MoE the problem is more subtle.

A MoE has three populations of parameters with very different characteristics: the dense backbone (small but updated every step), the experts (large but rarely activated), and the router (tiny but critical for routing). AdamW treats them all the same, assigning each the same amount of state. The paper “Where Should Optimizer State Live?” starts from an observation: the gradient statistics of these three populations are so different that treating them uniformly is wasteful.

SkewAdam assigns different precisions to the three components. The dense backbone gets full-precision state, experts get reduced-precision state, the router gets intermediate treatment. The result, measured on a 6.78 billion parameter MoE: optimizer state goes from 50.6 GB to 1.29 GB, a 97.4% cut. Total peak memory drops from 81.4 GB to 31.3 GB, bringing training within a 40 GB GPU budget with no measurable loss in validation perplexity.

The SLAI T-Rex paper tackles the problem from the other side: not the optimizer, but the infrastructure. Full post-training of DeepSeek-V4 on an Ascend NPU SuperPOD cluster requires managing memory pressure, communication overhead, and inefficient kernels. The paper documents end-to-end optimization on non-NVIDIA hardware, a detail that matters because the compute market is expanding beyond CUDA.

The two papers converge on one point: at trillion-parameter scale, memory is the main bottleneck, more than raw compute power. Optimizer state is the largest item in the budget, and distributing it well (with differentiated precision or redesigned system architectures) is what makes a cluster run instead of sit idle.

What remains open. SkewAdam is tested on a 6.78 billion parameter MoE, not trillions. The scalability of reduced precision at real scale is an open question. The SLAI T-Rex paper is engineering practice documented on specific hardware (Ascend), not a result generalizable to any cluster. Both papers are preprints on Hugging Face, without peer review as of July 23, 2026.