If you keep saying it “as a joke,” check whether you actually mean it.

A genuine joke gets made once and then moves on. Something said again and again in a light voice is usually a real opinion wearing a disguise that lets it be taken back if anyone objects. The format keeps you safe and lets the content out, which is the entire appeal of it.

Notice what you keep joking about and ask whether you would say it plainly. If you would, say it plainly, to the right person, and own it. If you would not, that is a reason to stop saying it at all.