Poor little guy! Poor you, this must be horribly stressful!
Kids, even smart ones, often don't know what they want because they don't have a basis of comparison. That is, he doesn't know how bad it is because, how could he? And it shouldn't be his responsibility to be able to judge his situation fairly (just like it's not his responsibility to know where an assembly is or what classroom to go to).
DD never wanted to change anything when she was young. She didn't want to be different, didn't want to stick out, didn't want to switch schools. But when we'd make the change for her, she ended up much happier.
I say your experience and broader knowledge has to supersede his more limited opinion. I say pull him out if you have any alternative.
(and as a nice bonus, you get to tell the school where they can stick their detention)
