I asked this question when I first went on a cruise. The consensus was that, when it comes to en route scenery, work out what side of the ship most of the land going past will be. Even then, you may well be too far out to see any land. If there is some spectacular scenic point, the captain will often do a 360 degree turn so both sides of the ship get a view.
When it comes to ports, there is no way to tell if port of starboard is best, because you don't know which way the ship will be facing when it docks. It might back into the dock, for example.