Why Systems Biology and Machine Learning shouldn’t work… but do… and what heat capacity teaches us about the mechanism of learning

In many fields, big data and large models are transforming scientific enquiry. These approaches usually depend on a very large number of ad hoc parameters that appear to be poorly specified. In spite of this complexity, these models are surprisingly predictive and appear to violate the canonical rules of statistics. To investigate the mechanisms of machine learning in this context, we develop a long-discussed analogy between statistical physics and inference in which sample size and inverse temperature are identified. This analogy suggests a novel relationship between the equipartition theorem and prediction that can explain many puzzling aspects of inference in applications ranging from our own biophysical experiments to the mechanism of learning algorithms like dropout. Finally, the approach suggests a new resolution to an old statistical problem: How to define an objective prior in Bayesian inference.​