Being first in your friend group to take something seriously can look uncool. Later it looks like a head start.

Groups quietly agree on how much effort is normal, and going past that line reads as a comment on everybody still standing behind it. The teasing is not really about you being odd. It is the group trying to pull its shape back. The thing is, effort compounds and teasing does not, so the gap you are opening now keeps widening long after anyone stopped joking.

You do not have to defend the choice. Keep the hours, keep the subject off the table when it only invites argument, and find one person outside the group who takes it as seriously as you do.