The person listening from the edge still belongs to the conversation’s effects.

Not speaking is not the same as not being in it. Whatever is said in a room lands on everyone present, and the effects do not distinguish between the people who contributed and the person who sat quietly at the end of the table. They carry it home in the same way, sometimes more, because they had no chance to shape it.

Notice who has not spoken in the next long conversation and find a way in for them that is not a spotlight. Ask afterwards, privately, what they made of it. If you are the quiet one, your reaction is a legitimate part of what happened.