People can enjoy your company while disliking the amount of freedom it requires you to keep.

Affection and acceptance are not always the same thing. Someone can genuinely love your company and still resent the solitude, the odd hours, the travel or the refusals that make you the person they enjoy, because those conditions cost them something. The wish is usually for a version of you with the freedom removed, which would not be the same person. Both feelings can be sincere at once.

If this is happening around you, try naming the connection out loud rather than defending each individual refusal. Explaining what the freedom is for tends to work better than arguing about any single evening.