From the sound of your post, it sounds as though you'd feel uncomfortable unless you dress up to please the formalists in the crowd. Then by all means, drag along the formal clothing to please them. It'll make their dining experience more enjoyable.
On the other hand, half the crowd will dress to please themselves by dressing casually with some possibly even wearing shorts.
It's your choice. Princess will allow just about anything lately.