The importance of being empty: Hopfield neural networks with diluted examples

We consider Hopfield networks, where neurons interact pair-wise by Hebbian couplings built over i. a set of definite patterns (ground-truths), ii. a sample of labeled examples (supervised setting), iii. a sample of unlabeled examples (unsupervised setting). We focus on the case where ground-truths are Rademacher vectors and examples are noisy versions of these ground-truths, possibly displaying some blank or empty entries (e.g., mimicking missing or dropped data), and we determine the spectral distribution of the coupling matrices in the three scenarios, by exploiting and extending the Marchenko-Pastur theorem. Building on this, we analytically inspect the generalization capabilities of the networks ii. and iii. examining their ability to recover ground-truth patterns from noisy inputs. In particular, as corroborated by long-running Monte Carlo simulations, the presence of blank entries can be beneficial in some conditions, suggesting strategies based on data sparsification; the robustness of these results in structured datasets is confirmed numerically. Finally, we demonstrate that the Hebbian matrix, built on sparse examples, can be recovered as the fixed point of a gradient descent algorithm with dropout, over a suitable loss function.