The phrase just asking can still carry an accusation.

Just asking presents a statement as a question and claims innocence in advance. The accusation is fully delivered, and the framing means that responding to it looks defensive while ignoring it looks like a concession. The disclaimer is doing work that the question alone could not.

If you are on the receiving end, answer the implication rather than the literal question, and you can say plainly that it sounded like more than a question. And if you catch yourself reaching for the phrase, consider saying the actual thing instead, which is more honest and easier to have a conversation about.