This is the 3rd week of KINDERGARTEN. Not knowing the whole story, of course, I believe kindergarten is about learning cooperation and caring. I would ask the teacher specifics about how your kiddo is "annoying" and how the behavior if the kids excluding him is different from bullying.
I think the word "bullying" has become overused. When I was growing up it would mean hitting someone or threatening to do so unless they gave their lunch money. Now it has been redefined to include "shunning". It is not possible to come up with an objective definition of "annoying". If a few kids in the OP's son's class find him annoying, how can they be proven wrong? If some children don't want to play with my kid, I'd tell him to play with other kids.
I was just suggesting that the OP investigate with the teacher how it's different. I don't have enough facts at this point, but really I think the teacher handled it poorly. Kids should indeed be learning to work things out, but as it sounds like there is immature behavior by both sides here, I think the teacher can do more than just say "he was being annoying, he needs to learn to deal with it." Maybe it was just poor communication by the teacher, and maybe the OP's kiddo was being annoying, but it still seems a bit harsh for the first couple weeks of school. More appropriate would be teacher intervention that teaches the kids better ways to handle the situation. (And again, I wasn't there, I was just reacting to the OP's first post.) But yes, I do think certain types of exclusion can be bullying. Not nice to encourage that in kindy.
ETA: And I think sitting in the classroom and observing is a fabulous idea, if you can do that without disruption.
