Not laughing at the mean joke makes you the odd one out for four seconds and the right one for years.

Group laughter has momentum, and the pressure to join is immediate and physical while the cost of joining is delayed and invisible. Four seconds of not laughing feels far longer than it is, which is exactly why the joke works. Nobody remembers the pause afterward. The person it was aimed at remembers who did not join in, and so will you.

You do not have to give a speech about it. Not laughing is enough, changing the subject is enough, and saying something to the person later is more than most people manage.