Equivalence testing and run-count design for stochastic machine-learning experiments
Provides rigorous statistical framework for ML researchers conducting comparative experiments with stochastic training methods.
AI Summary
Researchers develop statistical methods for equivalence testing in ML experiments, providing power theory and run-count formulas to certify practical equivalence between methods.
Excerpt
Claims that two training methods perform equally well often rest on a non-significant difference between two or three runs per method. We develop a protocol, with its power theory, for certifying practical equivalence in per-run metrics such as the AUROC (area under the receiver operating characteristic curve). It rests on a two-level (seed x item) variance model separating between-run dispersion from item-level noise. Equivalence within a pre-specified margin is tested by two one-sided tests (T