Is there a way you can compromise with the teacher? I think it's unreasonable to ask anyone to sit still and pay attention; if he has something to fidget with that doesn't bother others, I would ask her to accommodate that and I think it would be unreasonable for her not to.

I'm 30 and fidget all the time.

That said, I don't think you should pull him. He's old enough to learn to respect his teachers, and I think that you can advocate for his needs while still teaching him that he has to show his work because it's part of the class. He has to respect others. Pulling him teaches him that he has to power to exert over others.

I would not pull a child from a class unless I felt there were egregious behaviors on the part of the teacher.

you are right, it probably isn't a good fit. But he wont always have that, either. Teach your child to be better, even if it's painful... don't enable petulance.