A place can be wrong without being bad.

Not fitting somewhere is not the same as being somewhere terrible. Plenty of good places are wrong for a particular person at a particular time, and needing to leave does not require finding fault. People delay for years because they cannot build a case against a place that has done nothing wrong.

You are allowed to say it simply does not fit. That is a complete reason. It also lets you leave without bitterness, which makes it easier to visit, to stay connected, and to remember the place kindly.