As far as an experience of removing a child from a gifted program:
DS6s school has once a week pullouts for a couple of subjects. We pulled DS from the one he is furthest ahead in, as part of pulling him from that subject entirely (and homeschooling it). The range of abilities in the pull out was broad and the activities were not very different from in class activities. In addition it did not occur fully in the subject class time, but overlapped with something DS liked. DS is also limited by his slow writing, so we anticipated that when the pullout got around to doing anything written he would not find that rewarding. He has not missed it and the time we have gained at home is awesome.
In your DSs case I would just be sure he'd enjoy whatever his alternative is more, before pulling him.
