What you let slide in your friend group becomes the group’s actual rulebook.

Groups run on precedent rather than on stated values. Whatever gets laughed at, tolerated or left unchallenged becomes the standard, and everyone reads that standard accurately even though nobody writes it down. Silence in the moment reads as agreement, whatever you meant by it. That is how a group ends up with rules none of its members would defend out loud.

Next time something goes past the line, say one short thing instead of a speech. Even a quiet remark that it was harsh changes what the room believes is allowed. You are setting the rules either way, so it may as well be deliberate.