Statistical mechanics instruments in deep learning: three roles of entropy in neural networks

Modern deep learning systems are powerful, but their internal dynamics, modular structure, and dependence on representation scale remain difficult to understand and control. Ideas from statistical mechanics have long shaped neural-network theory, yet their practical use in modern deep learning remains uneven. This thesis develops entropy, a central concept of statistical mechanics and information theory, into a practical instrument for learning systems: a way to diagnose model behaviour, constrain learning objectives, and select representation scale. The thesis shows that entropy can play three connected roles across the learning pipeline. First, as a diagnostic tool, Entropy-Lens studies entropy profiles of intermediate token distributions in transformer language models, exposing structured expansion and pruning patterns during inference. Second, as a constructive principle, a maximum-entropy objective trains class-specialised modules that remain maximally uncertain outside their domain and can be composed into broader models. Third, as a scale-selecting inductive bias, spectral entropy of a graph diffusion process identifies a characteristic resolution scale for graph neural network learning, allowing graph representations to be rewired before learning and processed at multiple scales. Together, these contributions show entropy as a versatile practical instrument for deep learning. It compresses high-dimensional model behaviour into interpretable signals, turns off-domain uncertainty into a training constraint, and selects meaningful scales in relational data. More broadly, entropy helps decide what to read in a trained model, what to constrain during training, and at what scale to present relational data. The thesis argues that the relationship between statistical mechanics and deep learning is most concrete not as a complete theory of learning, but as a source of concepts that make specific aspects of learning systems measurable, constrained, and steerable.