AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
For researchers focused on overcoming catastrophic forgetting and improving continual adaptation of multimodal foundation models.
AI Summary
This arXiv paper introduces AlphaWiSE, a novel method for continual learning of models like CLIP, which adapts to new data without forgetting earlier cross-modal alignments. It proposes adaptive weight interpolation between frozen model checkpoints, optimizing a single scalar per parameter tensor using small exemplar memory. Experiments on audio-image-text retrieval show consistent improvements over existing continual learning baselines across multiple metrics.
Excerpt
Multimodal models such as CLIP learn a shared embedding space for cross-modal retrieval, but continual adaptation to sequentially arriving data can disrupt the cross-modal alignment acquired from earlier phases. Conventional continual-learning methods return a single checkpoint, which commits every retrieval direction to the same stability-plasticity trade-off. We propose AlphaWiSE, a post-hoc weight-space interpolation method that composes two frozen source checkpoints. For each aligned paramet
