← Back to feed

Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization

L5 · ResearcherResearcharXiv· 10/8/2026

Presents novel quantization techniques for reducing memory footprint in large-scale model training while maintaining optimization performance.

AI Summary

Researchers redesign 4-bit AdamW optimizer-state with ZIP-SR and ZE-EDEN methods, reducing validation-loss gap to 32-bit AdamW by up to 70% in GPT- and Llama-style models.

Excerpt

Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of \emph{rounding space}: the coordinate in which a quantizer chooses between adjacent reconstruction levels. For the second moment, a local analysis of the quantization cell adjacent to zero shows that small mean state error need not imply small mean

Read Original
0 upvotes · 0 downvotes · 1 min read

Related Articles

L5 · ResearcherResearcharXiv
VioLA: Learning Generalist Humanoid Control Policies from Human Data

VioLA trains humanoid control policies using human motion data by predicting motion latents that work for both humans and robots, achieving 100% zero-shot locomotion success on real hardware.

L5 · ResearcherResearcharXiv
FAITH: Feasibility-Aware Safety-Filtered RL for High-Dimensional Systems

Researchers propose FAITH, a feasibility-aware safety-filtered RL framework that achieves high safety rates while maintaining performance on high-dimensional robotic systems.

L5 · ResearcherResearcharXiv
From Reactive Containment to Proactive Assurance: Lessons from OpenAI, Anthropic, and Google Agent Security Incidents

Analyzes 2026 agent security incidents from OpenAI, Anthropic, and Google to propose a proactive security framework with boundary assurance stack.

L5 · ResearcherResearcharXiv
Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

Researchers developed white-box probes that detect LLM deception with up to 99.7% AUC, outperforming text-monitoring baselines.

L5 · ResearcherResearcharXiv
A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

Researchers introduce Success Guided Sampling (SGS) to improve massive-scale reinforcement learning for robotics by focusing training on task configurations at the policy's capability frontier.