The rest point is part of the route for the person who needs it.

Stopping is not an interruption of the journey when the journey requires it. People treat rests as failures of stamina, deviations from the real route, when for a lot of bodies the pause is a structural component of getting there at all. Planning around your own limits is not the same as giving in to them.

Build the stop into the plan for whatever reliably takes more out of you than it seems to, rather than hoping to get away without it. Say it out loud too, since somebody else in the group is probably wondering whether they are allowed to ask.