Imagine getting injured severely enough to warrant emergency medical attention and overnight hospital stay in someplace like Labadee Haiti and not having a passport. Now imagine trying to deal with being a “wealthy” American in their eyes and needing to rely on locals to help your loved ones get in touch with the US Embassy in Port au Prince (long way from Labadee) to try to get home since you don’t have a passport. Then there’s getting you to an airport capable of getting you to the US, etc.
If that doesn’t terrify you then you don’t know Haiti.
Get a passport.
ps…always be careful in foreign countries. Some despise Americans and the smiling faces you see at the port aren’t always sincere.