Quitting something that’s clearly not you isn’t weakness. Staying out of fear of looking like a quitter is.

Staying because leaving would look bad is a decision made entirely about other people’s reading of you. It costs years, and the people whose opinion you were managing are not paying that bill. Leaving something that was clearly wrong is information acted on, which is close to the opposite of giving up.

Ask whether you are staying for the thing itself or for how quitting would appear. If it is the second, set a date and go. Be honest with whoever is affected rather than drifting out slowly.