A wall you build to keep pain out is the same wall that keeps people from reaching you.

Protection is rarely selective. When you go quiet to avoid being hurt, the same quiet keeps out the people who would have been good to you, because there is no wall that only stops the dangerous ones. It works, in the sense that less gets in. That is also the cost, because less gets in.

You do not have to take the whole thing down. Let one person past one brick: tell a friend an actual thing that happened, not the summary you usually give. Watch what they do with it, and decide from there.