This actually is not a bad plan. You can check the SOL/Ellis Island boat schedule online.
However, instead of going to Penn Station just take the Path (take it from the World Trade Center Station) to Exchange Place and Uber/Taxi from there.
Downtown Manhattan (the Battery) can be very nice and extremely interesting to walk around especially if the weather is good.
If you are up to it, Manhattan is a VERY walkable city.
I am personally completely shocked that a luxury line like Silversea does not have a free transfer to Manhattan.
FYI, when you sail into NY harbor you pass directly under the Verrazano Bridge, you have a wonderful view of the lower Manhattan Skyline, and you will have a very nice view of the Statute of Liberty as you approach the port.