I think some of this is institutional culture. Take two small, competitive liberal arts colleges with similar stats on the surface, and one could have students whose main interests are Greek culture, preprofessionalism, skiing, etc and the other could have kids who are into--well, smoking pot, but also philosophy, film, neurology, arts performance, etc., and doing a lot of independent study.

I'm not saying I know anything about the school your son is at, but I do think even the small competitive schools vary a lot in terms of the kind of peers you get and I'm not at all sure it's as simple as IQ scores. I do think it's true that at a really large university, you may have a better chance of finding "your people," simply by virtue of size. It might take a long time, though.