Saying “no worries” when you’re actually worried just delays the conversation you’ll eventually need.

Waving something off buys quiet now and hands you the same problem later with more history attached. The other person takes you at your word and carries on, so when it does surface they are surprised and you look like you were storing it up. Nothing got resolved; the conversation was simply rescheduled for a worse day. Politeness and honesty are not really in conflict here.

When it does bother you, say the small version straight away, which is far easier than the big version in a month. One sentence will do: that it was not nothing, and what you would prefer. Then let it go properly.