If I were you I would meet with the teacher and ask her to be specific about how he is annoying the other boys. Hard to say if he is doing something wrong, or if they are getting away with bullying (could be a bit of both). Getting her to specifically pinpoint any behavior issues he has is important so you can work with him on those.
However, if the teacher can't get specific, and they continue with this, I would let the teacher know that you feel your son is being bullied. If she will not intervene, I would talk to the principal and let him know that you think the teacher is "soft on bullies"
Or something to that effect. Our school cracks down HARD from the principal's office when bullying is reported. Hopefully yours will do that same. However... our school would also not permit the exclusion to start with, so there could be a school culture problem there. Hard to know if it is the teacher or the school in general; by the end of this semester you will probably know which.
