Not everyone who sits beside you is traveling in your direction.

People can be near you for a long time without being headed anywhere similar. That is fine and often pleasant, and it becomes a problem only when you assume shared proximity means shared direction, and start making decisions on that assumption. Some companions are for a stretch of road. Expecting them to come the whole way is what creates the disappointment.

Be honest about who is with you for now and who is with you for the direction. Enjoy the first group without expecting the second thing from them. Most of the pain here comes from misfiled expectations rather than from anyone’s behaviour.