If standing firm costs you a few followers, they were only following the version of you that always agreed.

Some of the people around you are attached to your agreement rather than to you, which stays invisible until you disagree. When they go, the loss feels large and it is mostly a correction: you found out what the connection was made of. Whoever stays after you say something they did not like is a different and much more useful group.

If holding a position has cost you someone recently, resist reversing it to get them back. Check whether the position is still right, and if it is, hold it. Put your attention on the people who stayed.