The smartest move may look like nothing to someone who cannot see the board.

Good moves often look like nothing: the offer declined, the argument not entered, the position held while everyone else rearranges themselves. Their value lies in what they prevent and what they preserve, neither of which is visible to anyone without the full picture. So you will sometimes be judged as passive by people who are not seeing what you are seeing. That is simply part of the cost.

Stop measuring your decisions by how they look to observers. Ask whether the move improves your position over the timeframe you actually care about. Being misread is much cheaper than acting so that you are not.