Inside jokes are fun until you notice who’s standing there not laughing. Let them in on it.

An inside joke is a small fence. That is not a bad thing, it is how closeness works, and most of the time nobody is hurt by it. The trouble starts when someone is standing right there watching a wall go up in real time. They are not upset about the joke, they are doing the maths on whether they belong.

When you catch that look, the repair takes about six seconds. Explain the joke badly, with the whole boring backstory. The joke gets less funny and the person gets to stay in the group, which is a good trade.