Some people only recognize your growth as distance because they stayed where you left.

When you change and someone else does not, the space between you increases, and from where they are standing it looks like you moved away from them. In a literal sense you did, though not as a rejection. They are describing an accurate observation with an inaccurate explanation. Understanding that makes the conversation easier to have.

Stay in contact where you can, and do not require others to change in order to keep them. Some distance is simply structural. Where the relationship matters, make the effort to bridge it rather than waiting for them to.