I had an interesting situation on a British Isles cruise over the summer. I changed out of my excursion clothes (capris and a t-shirt) before dinner. I put on dress pants, nice shirt, and a blazer. It was "formal" night and this was something I'd generally wear to work.
After an early dinner, I changed into a hoodie and long, casual pants because I wanted to go outside on the deck and it was starting to get chilly. I have mobility issues and don't like walking outside in my heeled dress shoes. I didn't go back to my stateroom and instead went direct to a bar to hear a pianist. An older lady walked by me and my husband and said I guess you don't have appropriate clothes for formal night!
My point is that even if you do dress "appropriately" for dinner, you may still get a snide comment outside of the MDR if during the dinner period. Just live and let live!