The relationship defined as casual can still carry very formal expectations that nobody wrote down.

Calling something casual does not remove the expectations; it removes the ability to state them. Both people still have assumptions about frequency, exclusivity, and what counts as reasonable, and the label makes raising any of that seem like a violation of the terms. Informality is not the same as an absence of rules.

If something in your life is officially casual and privately not, it is worth saying so once, plainly, and finding out where you both actually stand. The conversation is uncomfortable and it is much shorter than the alternative, which is discovering the unwritten expectations one by one as each of them gets broken.