You can decline an activity while remaining interested in the person who invited you.

A no to a plan is easily heard as a no to a person. Turn down enough evenings and, without anything being said, the other side concludes you are drifting away, since declining is the only signal they have to go on. The two are genuinely separate. Not wanting the loud restaurant says nothing about wanting the friendship.

Decline the thing and name the person in the same breath. Say the specific reason it is not for you, then propose something you would actually like. Doing the second half is what keeps the first half from being misread.