If the major obstacle is the classroom schedule, then I don't see any reason to hold him back. Class schedules at that grade level should be pretty flexible. We've run into scheduling conflicts for my DD with her GT pullouts (two per day, and much longer than an hour), and once I've reported them to the proper authorities, adjustments were made.

That leaves the social concerns. What does your DS say about these?