Is the school wanting to hold him back just because of his age? If so, then I wouldn't go along with that. Or do they have reasons beyond age?
I suspect maybe we need more information than you're giving us... Or do you really just have two relatively unsupported opinions, without much in the way of supporting evidence? If there's nothing more than opinion, then I'd say you need to do some more talking to find out why the dr. and the school feel as they do. Details matter in this sort of situation!
Regardless, I wouldn't place my child in an inappropriate school setting just so he could have a second recess. I'd put that decision in the "doing what's good for you whether you like it or not" pile, right there with eating your veggies and taking your medicine.
