“Sorry you feel that way” is a return label slapped on someone else’s pain.

The sentence is shaped like an apology and does the opposite job, because it puts the problem in their reaction rather than in anything you did. People hear that instantly, even when they cannot explain why it landed badly. It closes the exchange without repairing anything, and usually leaves them more alone with it than before you spoke.

Use the real version instead, which has two parts: what you did, and that you regret the effect. I said that in front of everyone and it was unfair. No conditions, and no mention of how they took it.