If your joke only works when someone smaller is the target, it’s not funny. It’s just safe for you.

Punching down is easy because the target cannot answer without making it worse for themselves. That is what makes the room laugh quickly and also what makes the laugh thin. Real comedy takes a risk somewhere, usually with yourself. If the whole mechanism is that someone has less power than you, it is not a joke, it is just safe.

Notice who the laugh is costing next time, and if it is one person repeatedly, stop. Aim the next one at yourself and watch that it still works.