Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs
Presents novel theoretical framework for graph neural network compression that could influence future model deployment strategies.
AI Summary
Researchers propose G²MLP, a graph geometry-aware MLP distillation method that addresses spectral failure modes in GNN-to-MLP knowledge transfer using Ollivier-Ricci curvature.
Excerpt
GNN-to-MLP distillation aims to retain the predictive accuracy of a message-passing teacher while deploying a graph-free MLP at inference. Existing methods mainly transfer node-wise predictions or use confidence-based reweighting, but they do not specify where the student should preserve the teacher's graph-induced geometry. We show that this omission leads to two spectral failure modes in the student's representation space. On sparse graphs, the student suffers from spectral underfit, missing h