A simple invitation is easier to accept when declining it will not become another problem.

How easy an invitation is to accept depends mostly on what a refusal would cost. If declining brings follow-up questions, visible disappointment, or a slightly cooler tone next time, the invitation is really a small obligation with a friendly face on it. People then protect themselves by going quiet instead of answering. An easy exit is what makes the door easy to walk through.

When you ask, say plainly that either answer is welcome, and then behave that way. Do not ask why, and do not mention it later. Invite them again next time regardless of what they said.