Your little brother is annoying and also someone God is crazy about. Both are true. Pray accordingly.

Both facts about your brother are true at the same time and neither cancels the other. He is genuinely irritating, you are not imagining it, and he is also somebody entirely loved outside your assessment of him. Holding both is harder than picking one, because picking one lets you either be furious or pretend to be fine. Keeping them together is what stops annoyance from hardening into contempt.

Pray for him by name this week while still finding him annoying. No pretending. Then do one quiet decent thing for him that he does not find out about.