You scraped their bumper. Knocking on the door is scary. Driving away makes you somebody else.

Fear in that moment is real and is not the problem. What happens next is a fork that quietly decides which version of yourself you keep, because driving away solves the afternoon and installs something permanent: a piece of knowledge about what you do when the cost lands on you. The scrape is minor and repairable. The other thing is not.

Leave the note. Say what happened without softening it, and give a number where you can be reached. It will cost money and one unpleasant conversation, and then you will not think about it again.