The way I like to explain the reasoning is that these are not fees that the ports are charging to the passengers. They are fees that the ports are charging to the cruise line for docking there. The cruise lines are just adding additional revenue on top of the cruise fare to try to cover these expenses.
The amount they add is just an estimate based on their expected occupancy for the cruise. If the cruise does not sell as well as expected, these fees might come out to more than they collected. They don't go back and charge the passengers for the difference. It also means that if they miss a port, the amount collected might still be less than the actual overall expenses they are charged. If the cruise sells better than expected, the cruise lines might add OBC to your account.
Since these are not fees that are charged directly to the passengers, the cruise lines have no obligation to refund/credit passengers.