Fate of Lorentz symmetry in relative-locality momentum spaces