Silence feels different when you know somebody is glad you are there.

The absence of talk carries whatever the room already contains. Sitting quietly beside someone who wants you there is restful, and the identical silence with someone indifferent is work, because you spend it wondering what it means. Nothing audible distinguishes them. The difference is entirely in what you believe about being welcome.

With the people closest to you, say the welcome out loud occasionally so the quiet has something to rest on. Glad you came, or nice having you here. Then let the silence happen without filling it, and see how much easier it is to stay in.