People with nothing to prove are the hardest people to manipulate.

Manipulation needs a handle. It works by finding what you need to be seen as, then threatening it: not tough enough, not loyal, not fun, scared. Somebody with nothing to prove has no handle to grab, so the dare just sits there and nothing happens. That is why the calmest person in a group is usually the hardest one to push into something stupid.

Find your handle. Notice which insult would actually move you, or which dare you would take rather than be seen refusing. Naming it takes most of its power, and next time you feel that exact tug you will recognize what is being pulled.