Telling someone “that joke bothered me” is a boundary, not an attack.

Saying that something landed badly is a piece of information, not a charge. People hear it as an accusation mostly when it arrives with heat, so the shorter and calmer it is, the easier it becomes to take. You are reporting your own reaction, which nobody can argue you out of. What they do with it afterward is theirs.

Try one plain sentence the next time it happens, in the moment and without a lecture. Say the joke did not sit well with you, then let the conversation move along.