Everybody has an accent. Some people just haven’t left home long enough to hear theirs.

There is no neutral way of speaking, only the way that matches the room you happen to be standing in. People who have never been the odd voice somewhere mistake their own sound for the default. That is not a fact about language; it is a fact about how little distance they have covered.

When somebody comments on how you talk, you can treat it as information about their travels. Ask where they are from and listen for their own vowels. Keep your voice where it is; the inconvenience belongs to the person who has only ever heard one.