Could he do math independently this year with AoPS or something else that might be appropriate?
If he is happy socially in his current placement, I'd be very hesitant to eliminate the only times during the day where unrestricted social interaction occurs. Frankly, for HG+ (and I don't know that this is where your son falls--I'm just assuming given the typical audience of this board) kids, my opinion is that the social needs should trump all others given that it is very unlikely that intellectual needs will be met in school. Of course, for some kids, intellectual needs and social needs are strongly intertwined, but for others, and your son sounds like he might be one of them, they are uncoupled enough that they do well with age peers.
