The people who attend quietly still belong in the plan for next time.

Planning tends to be shaped by whoever speaks. Loud preferences get accommodated, and the person who arrives, enjoys it, and says little is assumed to be satisfied with whatever happens, so their needs never enter the arrangement at all. Quiet is easy to read as agreement and is often just a reluctance to make demands in front of a group.

Before the next gathering, ask one or two of the quieter people what they would like, privately and specifically. A direct question in a small setting gets honest answers that a general one never will.