Instability of optical solitons in the boundary value problem for a medium of finite extension