Many adults can't deal with a room full of crying two year olds and find it stressful. The goal of socialization is a long term goal and that environment wouldn't be representative of what your son would experience around older children. And it is not fun or useful for a bright kid to sit around hearing stuff repeated below their abilities.

Don't let society's "oughts" overcome your parental instincts.