Disappointing someone isn’t the same as wronging them.

Letting someone down and treating them badly get felt as the same thing, and they are not. Disappointment is what happens when your answer differs from their hope, which occurs constantly between decent people. Wronging someone involves a broken promise or a harm. Collapsing the two makes every no a small crime, and you end up agreeing to things to avoid being bad.

Next time you decline something and feel guilty, ask which of the two it was. If you kept your word and were honest, the disappointment is theirs to feel and yours to allow. You can be sorry it is not what they wanted without being wrong.