You can return to the same location without returning to the same life.

The map stays put; the life does not. You can move back to the exact address and find that the friends have scattered, the jobs have changed, and the version of you that fitted there no longer exists. The disappointment comes from expecting the place to restore a situation it never actually contained.

If you are drawn back somewhere, be specific about what you are hoping to find. If it is people or a period rather than a location, going there will not deliver it, and knowing that in advance saves a painful discovery.