Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks (i.e. Hopfield networks with polynomial interactions of even degree P > 2), the purpose of this paper is twofold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of replica symmetry breaking (RSB) in these networks, then -deepening results stemmed via these routes- we aim to inspect the glassiness that they hide. In particular, regarding the methodology, we provide two techniques: the former (closer to mathematical physics in spirit) is an adaptation of the transport PDE to this case, while the latter (more probabilistic in its nature) is an extension of Guerra’s interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner’s picture (heuristically achieved through the replica trick) and we identify the maximal storage capacity by a groundstate analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: at difference with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase.We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network (and, in a cascade fashion, all its properties) as a linear combination of the free energies of a hard spin glass (i.e. the Sherrington–Kirkpatrick model) and a soft spin glass (the Gaussian or ”spherical”model). While this continues to hold also in the first step of RSB for the Hopfieldmodel, this is no longer truewhen interactions are more than pairwise (whatever the level of description, RS or RSB). For dense networks solely the free energy of the hard spin glass survives. As the Sherrington–Kirkpatrick spin glass is full-RSB (i.e. Parisi theory holds for that model), while the Gaussian spin-glass is replica symmetric, these different representation theorems prove a huge diversity in the underlying glassiness of associative neural networks.

Replica symmetry breaking in dense Hebbian neural networks / Francesco Alemanno, Linda Albanese.; Alessandrelli, Andrea; Barra, Adriano. - In: JOURNAL OF STATISTICAL PHYSICS. - ISSN 0022-4715. - 189:(2022). [10.1007/s10955-022-02966-8]

Replica symmetry breaking in dense Hebbian neural networks

Adriano Barra
2022

Abstract

Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks (i.e. Hopfield networks with polynomial interactions of even degree P > 2), the purpose of this paper is twofold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of replica symmetry breaking (RSB) in these networks, then -deepening results stemmed via these routes- we aim to inspect the glassiness that they hide. In particular, regarding the methodology, we provide two techniques: the former (closer to mathematical physics in spirit) is an adaptation of the transport PDE to this case, while the latter (more probabilistic in its nature) is an extension of Guerra’s interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner’s picture (heuristically achieved through the replica trick) and we identify the maximal storage capacity by a groundstate analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: at difference with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase.We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network (and, in a cascade fashion, all its properties) as a linear combination of the free energies of a hard spin glass (i.e. the Sherrington–Kirkpatrick model) and a soft spin glass (the Gaussian or ”spherical”model). While this continues to hold also in the first step of RSB for the Hopfieldmodel, this is no longer truewhen interactions are more than pairwise (whatever the level of description, RS or RSB). For dense networks solely the free energy of the hard spin glass survives. As the Sherrington–Kirkpatrick spin glass is full-RSB (i.e. Parisi theory holds for that model), while the Gaussian spin-glass is replica symmetric, these different representation theorems prove a huge diversity in the underlying glassiness of associative neural networks.
2022
Hebbian neural networks; Replica symmetry breaking; Pattern recognition
01 Pubblicazione su rivista::01a Articolo in rivista
Replica symmetry breaking in dense Hebbian neural networks / Francesco Alemanno, Linda Albanese.; Alessandrelli, Andrea; Barra, Adriano. - In: JOURNAL OF STATISTICAL PHYSICS. - ISSN 0022-4715. - 189:(2022). [10.1007/s10955-022-02966-8]
File allegati a questo prodotto
File Dimensione Formato  
Albanese_Replica_2022.pdf

accesso aperto

Tipologia: Versione editoriale (versione pubblicata con il layout dell'editore)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 865.14 kB
Formato Adobe PDF
865.14 kB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11573/1707784
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 9
  • ???jsp.display-item.citation.isi??? ND
social impact