Guilt shows up when you change the rules. That doesn’t prove the new rules are wrong.

Guilt is a reaction to breaking a pattern rather than a measurement of whether the pattern was good. Change what you do and the feeling arrives on schedule, because part of you is tracking the old arrangement and reporting a deviation from it. The timing gives it away: it turns up with the change, not with any actual harm.

When it appears after a reasonable no, check for damage instead of checking the feeling. Ask who was hurt, and by what. If the answer is nobody, let the guilt run its course without rewriting the decision, because it does fade, usually within a couple of weeks.