You’re not the same person who made that mistake. Stop paying their bill.

Guilt tends to outstay its usefulness. It works for about as long as it takes to understand what went wrong and repair what can be repaired, and after that it is interest on a debt already settled. Meanwhile the person who did the thing has gone on changing, which is what makes the continued payment strange.

If something can still be made right, do that part properly, then stop. Apologize once and do it well. After that, let the evidence of who you are come from this month rather than that one.