Someone trying to get a rise out of you is asking you to hand over the controls. You can decline.

Somebody poking at you has found out which button works, and your reaction is the payment they are collecting. The content barely matters; the point is watching you move. Once you see it as a transaction rather than an insult, you get to decide whether to pay, and you are allowed to decline without explaining yourself.

Pick a default for the next time: a flat okay, or saying nothing and carrying on with what you were doing. If it keeps happening and starts to wear on you, tell somebody you trust rather than carrying it alone.