A person who values your presence will notice when their behavior keeps creating your absence.

Someone who actually wants you around will eventually connect their behaviour to your withdrawal, because they are paying attention to the effect they have. When the same pattern repeats and the connection is never made, the absence of curiosity is itself the information. You can point it out once. Whether they take it up is not something you can arrange.

Say clearly what causes you to pull back rather than simply pulling back. Then watch whether anything changes. If the pattern continues unexamined, you have your answer without needing a confrontation.