Yes, I think it pays to remember that any sort of educational "weirdness" on your part is a lot more important to you than it is to anyone else. The more matter-of-fact and comfortable you can be, the better. If you seem defensive, you invite them to go on the offensive, which you do NOT need. If you can perfect your casual shrug, you'll be fine.
I think it's Grinity who recommends practicing your answer in front of the mirror until it feels casual and easy to give. (Or was that you, Dottie? My memory is so rotten...) Regardless of who said it, I think it's great advice!
We had the same sort of problem with giving a response about why we were homeschooling, and my usual answer is to say, "Well," and then lower my voice to say, "he's pretty bright," and then back up to normal volume to deliver "and he just couldn't get what he needed there."
That's it.
No blame. No grandstanding. No further explanation. Just honesty delivered as simply and subtly as possible. The lowered voice and use of the phrase "pretty bright" seems to communicate to people that this isn't something you're bragging about or making much of.
You could do the same sort of thing with grade-skipping, only change the end to "and he just couldn't get what he needed in Xth grade." Parents understand the importance of giving kids what they need, and it shows that you considered well the "normal" way first, that you aren't one of "those" parents. (Whoever "they" are...)
But mostly, be relaxed and casual about whatever you say. Your attitude matters more than any words you say, I think.
HTH!
