In my experience, kids who do well with age mates also do well in other environments because of social skills. And kids who struggle socially may struggle in many environments, not just the age-based classroom.
I agree. If a gifted kid is struggling socially with age mates, there is a temptation to blame it on the GT when it really could be something else - not just that they are too intellectually different. I know several PG kids who genuinely enjoy and are comfortable playing with same age peers (DS included). That said, when a teacher makes a child stand out by doing things like teaching other kids or otherwise not being sensitive to the differences in ability it can be a pretty bad social fit.
