If you were part of the joke that went too far, “I didn’t start it” isn’t an exit.

Being the second or fourth person to laugh is still part of what made it happen. Jokes need a room, and the room is made of people who joined in, which means nobody on the receiving end was hurt by the person who began it alone. Not starting it is accurate, and it is not the same as not being involved.

If you know you were in that room, you could say so to the person, briefly and without a speech. Not a confession to being the worst one, just that you were there and went along.