If they’d only say it while your friend is in the bathroom, that’s your cue to say something back.

Timing gives away intent. A remark saved for the moment someone cannot answer it is not an observation, it is an invitation to agree with something the speaker would not defend to their face. Your silence gets counted as agreement, which is usually the point of the timing.

You do not need a confrontation, just a short line that declines the invitation. Say you would rather not, or mention that you disagree, and then move the conversation on. It is over in four seconds, and the person learns that you are not an audience for that.