Let them talk. People who spend all their time discussing you are spending none of it catching up.

Talking about someone is a full time occupation for the person doing it. Every hour spent tracking what you did is an hour not spent on their own work, and they rarely notice the trade. Responding pulls you into the same accounting, because now you are spending your hours on them too.

When you hear that people are discussing you, let the information land and then go back to what you were doing. Do not audit the rumor or correct the record. Keep moving, and let the distance between what you are building and what they are saying widen on its own.