The long term fear will of course be that you started him "too early", but keep in mind that this is more likely personality.
I wish people would understand that it's okay to be an introvert. Not everyone babbles non-stop, and that's okay. Thinking before speaking is actually a desirable trait. Albeit a rare one.
I agree, it's a personality trait and there is nothing wrong with that. I was this way and so are my kids. Not to mention that after a while most of the questions don't seem to be worth answering. One cannot sometimes even believe that a teacher would ask such easy questions.
