It almost seems that one, or both of you, is determined to push the boundaries as low as you can.
Why not turn it around and see how much nicer you'd both look completely "glammed up".
Then, you'd not look out of place and probably love being part of the experience. It doesn't have to cost a fortune, but I'm sure, you'd have a far better time if you respected the Cunard traditions, and feel more part of the "family" if you chose to aspire to a higher dress appearance and enjoy the ambiance.