Detecting stable-unstable nonlinear invariant manifold and homoclinic orbits in mechanical systems