Chiming in late- I put my son with his agemates, as he is not a big kid and not particularly athletic, though he enjoys sports. It worked out well for the most part, with many of the kids in his den also playing on the same sports teams. The kids who stick with scouting tend to be high achievers/bright/gifted or else they have parents who are very involved and supportive.
My son is a first year boy scout now, and he is getting lots of positive feedback from the leaders, both the kids (it's a boy run troop) and the adults. I think itis a great program, though of course YMMV based upon the leaders and boys in each pack or troop.