You can be the funny one and still have very serious dreams.

Roles get assigned early and then quietly enforced. Once a group has decided you are the funny one, ambition from you reads as a bit, and making the joke is easier than watching somebody look confused. The role was shorthand though, never a full description, and you were more than one thing before anyone labeled you.

Say the serious thing once this week with no punchline attached to the end. It feels exposed for about four seconds. The people worth keeping will adjust, and the ones who cannot are telling you something useful for free.