Prediction Is Not a Sufficient Memory State
Identifies fundamental limitations in memory representation for ML systems, relevant to theoretical ML research.
AI Summary
Theoretical paper proves that prediction alone is insufficient for memory in learning systems, requiring additional state information for exact cumulative learning.
Excerpt
A memory that reproduces every current prediction may still have forgotten how to learn from the next observation. We characterize this distinction for exact cumulative learning and attention. For smooth convex residual losses with unrestricted real features and labels, a continuous recurrent state of fixed dimension exists exactly when the loss is polynomial. With ordinary, unweighted observations, degree $D$ requires precisely $r=\binom{d+D}{D}-1$ coordinates; even among histories with one fix