Whispering in front of someone feels private to you and like a verdict to them.

From inside a whisper you are simply having a small conversation. From outside, somebody can see two people deciding something and can hear none of it, so their mind fills in the content, usually with themselves as the subject. What was said barely matters; the shape of the thing is the message. In a room of three there is only one available interpretation.

If it has to be private, take it somewhere else or wait. If it does not, say it out loud. And if you catch yourself mid-whisper in front of a third person, tell them what it was about.