Sunnyday, not trying to imply my kids are bright or yours are not - my kids and I are "Not So Gifted"!
My kids were far from the brightest in their HS classes. They did a variety of activities and classes and did not attend elite schools for undergrad. They did not take classes outside of their HS. We did have some concerns that they were not being challenged, and in the case of one kid, that was probably true.
However, they turned out just fine. Or are headed in the right direction, as both older ones are back in school. I don't know that there is a right or wrong answer for your son - chances are high that he will turn out just fine no matter which path you choose.
If he wants to try for the selective school, he should go for it. However, it should be because he wants to do it. I have tried to push my kids toward certain choices at certain times, and it didn't really turn out so well.