Learning Hyperparameters with Variational Bayes: The Case for a Data-Emphasized Objective

Abstract: A basic problem in supervised machine learning is avoiding overfitting when training a big neural network classifier. To solve this today, practitioners often resort to grid searches to tune model hyperparameters like regularization strength. This can work, but often takes many hours. To be more efficient, we suggest revisiting a classic idea from Bayesian statistics: learning hyperparameters to optimize the marginal likelihood. This method can avoid overfitting via an inherent Occam’s razor while replacing the expensive grid search with one run of gradient descent. However, it is only tractable for small-scale models. For big neural nets, common variational approximations produce inferior hyperparameters and poor fits. We’ll demonstrate how a revised variational objective that emphasizes the data makes gradient-based learning of model hyperparameters possible. Practitioners can then achieve well-tuned ResNet or vision transformer classifiers in just a few hours, avoiding the 80+ hour grid searches common in recent literature.

Bio: Michael C. Hughes (“Mike”) is an Associate Professor of Computer Science at Tufts University. His group does research in statistical machine learning and its applications to healthcare. His research interests include Bayesian modeling, learning from limited labeled data, decision-aware learning, and optimization algorithms for approximate inference. You can find his papers and open-source code on the web at www.michaelchughes.com.