Any chance your DS is shy in class and doesn't speak up because he feels different than the other kids and is hiding his abilities? My DD was like that and teachers were concerned about it. That is, until she skipped a grade and started speaking up because she no longer felt the need to hide herself. She still does that in math a bit, but only because she's 11 taking Algebra and still finds it much easier than all her classmates, and she doesn't like for people to know that because it makes her stand out. In the classes in which she feels she fits in, she participates just like anyone else. Any chance that this is partly what is going on?
That being said, shyness and introversion are usually innate traits that tend to hold over time. Which is not to say that there are not ways to improve their expression (e.g., practice speaking in front of people, practice with different social situations, making the environment more comfortable). It can be hard to be an introverted person in an extroverted world. And it can be especially hard to be an introverted kid in an educational system generally run by extroverts (afterall, most teachers are extroverts simply for the fact that they enjoy getting up in front of people for hours every day).
Also, I'd like to say that the fact that your DS tends to gravitate toward younger kids and girls might have something to do with his possibly being a more gentle soul than many of the other boys. My DS has always been this way, from toddlerhood to his teenage years. Girls, in particular, tend to be slightly calmer than boys at that age, and my son has always liked that, for lack of a better word, predictability.
It seems to me that you are comfortable with the idea of him going to first this year, it's just the school who is wary. IMO, the idea of making him repeat K is a nonstarter.
I would be curious to know if his pre-K teacher has a different opinion of your DS's shyness than his K teachers.