Having your number doesn’t mean someone is entitled to your time.

A contact is a way to reach you, not a claim on you. The two got confused somewhere, so an unanswered message starts feeling like a broken promise instead of something you simply have not gotten to yet. Nobody is owed an immediate reply just because sending the message was easy on their end.

Answer on your own schedule and stop apologizing for the gap. If somebody uses access badly and keeps doing it, you can mute, delay, or stop replying without explaining yourself. The list of people who get fast answers from you should be short, and you should be the one who wrote it.