This is the 3rd week of KINDERGARTEN. Not knowing the whole story, of course, I believe kindergarten is about learning cooperation and caring. I would ask the teacher specifics about how your kiddo is "annoying" and how the behavior if the kids excluding him is different from bullying.
I think the word "bullying" has become overused. When I was growing up it would mean hitting someone or threatening to do so unless they gave their lunch money. Now it has been redefined to include "shunning". It is not possible to come up with an objective definition of "annoying". If a few kids in the OP's son's class find him annoying, how can they be proven wrong? If some children don't want to play with my kid, I'd tell him to play with other kids.
