You can dislike a stereotype while quietly wanting to be its flattering exception.

It is easy to reject a generalisation while wanting to be the flattering exception to it. The wish to be told you are not like the others accepts the category and only asks for personal exemption, which leaves the thing you objected to fully intact. The compliment and the prejudice are made of the same material.

When someone tells you that you are not like the rest, notice what that sentence assumes about the rest. You can decline it gently and specifically. Refusing the exemption is the only version of the objection that actually costs you anything.