While we haven’t arrived as late as you, more like 1 pm, we felt the same way….what to do?! We explored some foodie excursions but at the end of the day, we ended walking through town and up to the fort. Toured it, took some pics, and headed back to the ship. You may get some more insight by visiting the ports of call page but were in your boat, I’d probably stay on board or at the very least, walk around a bit and then head back to the ship.