Mooring system optimization for vessel in waves