I know what you mean about pulling him out having negative side effects. I would really only do that as a last resort (i.e. leaving him in would be MORE damaging.)
See what I mean? How damaged would he have to be before you pulled him? And for what purpose? Just so he can be like everyone else? He isn't. Not everyone is. That is how humans are. We vary. My homeschooling contacts say it is so much easier to get them before they have been damaged - you can always sent him later. Please don't let the priority list thing scare you out of doing what is right. You never know - perhaps the fartherer away school has the just right best friend sitting right there waiting for him in his future classroom.
Actually, are the individual children moved every two weeks. My DS would have loved that, actually! One of my early ideas was to rotate him through all the teachers so he wouldn't have a chance to get bored. (joking)
Bests -
Trinity