Computational issues in Optimization for Deep networks

In this paper, we aim at investigating some relevant computational issues, in particular the role of global vs local optimizers, the role of the choice of the starting point, as well as of the optimization hyper-parameters setting on classification performances of deep networks. We carry out extensive computational experiments using nine different open-source optimization algorithms to train a deep CNN on an image classification task. Eventually, we also assess the role of wideness and depth on the computational optimization performance, carrying out additional tests with wider and deeper neural architecture.