The person reading silently is still part of the audience your words may affect.

A room contains more people than the ones talking. The quiet person at the end of the table, the new colleague, the one who has decided this is not a place to speak are all forming an impression of what is safe to say here, and a remark you thought was between two people is working on all of them. Silence is not absence. Whoever says nothing is often most affected by the tone.

Look at who has not spoken and ask them directly what they think. Address the room as though the quietest person’s view of it matters, because it usually does.