Shellymos,
Your son sounds so much like mine!
He noticed he was different (or at least the first time he verbally expressed his observation) was when he was 4 years old, and he asked, "Do any other 4 year olds know how to do double-digit by single-digit multiplication like me?" I told him, "I'm sure there are," but in my mind I'm thinking, yeah, but I have no idea WHERE they are (until I found this board, when I realized that they actually DO exist - lol!)
My son was actually "grade-skipped" in preschool - three weeks into the 3 year old preschool class, the assistant director of the school called me into her office and said that she and his teacher thought it might be better for my son if we moved him to the 4 year old preschool class. He would be 4 anyway in a month, so we waited till after he turned 4 and moved him up. Interestingly enough, the kids in his 4 year old preschool class clearly noticed that my son was "different". When I was volunteering in the classroom one day, the class was at circle time and one little boy asked, "Is he in first grade? How does he know all this stuff?" (i.e., letters and their sounds, numbers). The teacher was wonderful at her explanation, saying that my DS really loves numbers and letters, so he works and plays with them a lot, just like how the little boy who asked the question really loved cars and trucks so he knew all about them.
DH and I have explained to our son that everyone does things at their own pace. Some people learn things faster than others, but everyone is always learning. And some people are better at things than others. He loves to learn, is very good at it, and he knows he is good at math, and we also tell him that Mom is good at xxx and Dad is really good at yyy, isn't it neat how people have different interests and are enjoy different things - it makes the world a more interesting place.
I do believe this conversation with DS will evolve as he gets older. Especially as he goes to school (starting K with the hope of skipping to 1st after an assessment period).