← Back to feed

Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping

L5 · ResearcherResearcharXiv· 10/7/2026

Provides theoretical foundation for distributed ML optimization under realistic noise conditions

AI Summary

Researchers prove that clipped decentralized SGD achieves optimal convergence rates under heavy-tailed noise conditions while maintaining linear speed-up across agents.

Excerpt

Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization. While these methods are well understood in centralized settings, much less is known in decentralized ones, where applying a nonlinearity to local gradients affects both optimization and consensus. Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal

Read Original
0 upvotes · 0 downvotes · 1 min read

Related Articles

L5 · ResearcherResearcharXiv
Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

Researchers propose G²MLP, a graph geometry-aware MLP distillation method that addresses spectral failure modes in GNN-to-MLP knowledge transfer using Ollivier-Ricci curvature.

L5 · ResearcherResearcharXiv
Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

Research paper identifies systematic language sensitivity in vision-language-action models where minor phrasing changes cause performance swings up to 61 points, and proposes rule-based rephrasing method that improves success rates by 16-27%.

L5 · ResearcherResearcharXiv
Decoupling Exploration from Optimization in RLVR

Researchers propose Exploration-Distillation (ExpDis), a method that decouples exploration from optimization in RLVR training, improving mathematical reasoning performance across seven benchmarks.

L5 · ResearcherResearcharXiv
EngramEdit: Decoupled Knowledge Updates in LLMs through Conditional Memory

EngramEdit enables precise LLM knowledge updates via conditional memory, achieving 3x baseline accuracy in chain-of-thought reasoning while preserving unrelated knowledge.

L5 · ResearcherResearch@emollick.bsky.social
Ethan Mollick (@emollick.bsky.social): Some early first-hand accounts of the experience of encountering a narrow superhuman intelligence as mathematicians grapple with the hundreds of big AI proofs released by OpenAI. Problems solved in…

Mathematicians are struggling to understand 372 AI-generated proofs from OpenAI, including a proof of the long-unsolved Unique Games Conjecture, which appear superhuman but incomprehensible to humans.