Chasing passive galaxies in the early Universe: a critical analysis in CANDELS GOODS-South