I am guessing the teacher is spot on when she says he is annoying the boys. He doesn't handle peer rejection well. While he isn't aggressive when his feelings are hurt, he turns pesky - l am guessing it is things like flipping a shirt collar as he walks by, putting a foot on them under a table, bumping into them "on accident".
DH pointed out that this is likely being seen as the social/emotional immaturity schools warn of when considering early K. Perhaps if he were older, he would be able to walk away with his feelings intact when other boys don't want to include him. But maybe not. He is sensitive and thinks about these things WAY too deeply and probably always will. Since pick-up this afternoon I have fielded these questions "Why do you love me? Like what about me makes you love me?" and "Did you want to have kids? What about a kid like me that does things that people don't like or think is weird? Do you think I do those things because I was born too early? (he was premature) You know how things are messed up sometimes if you don't wait long enough. Like food in an oven or an experiment or something. Do you think I am like that?".

(sigh)...off to look at "dream schools" and threads that discuss homeschooling while working full-time.