On the space of connections having non-trivial twisted harmonic spinors