Not all criticism is feedback. Some of it is a complaint about losing access, dressed up as concern for who you used to be, and it usually appears right after you start saying no. The person is not describing you, they are describing the effect on them.

When you hear it, ask yourself one honest question: have I become unkind, or just less available. If it is the first, fix it, because that is worth fixing. If it is the second, you are allowed to keep going.