It's the first year jitters. Give it a few months and this statement will no longer bother you. It really is an equivalent of "How are you?" You will get there too, trust me.
I think most of the people feel this way at first when they are put into a situation that it's out of the norm. It's not only homeschoolers but also parents who accelerate their children, send them to college early, etc.
When we started homeschooling it took me a few months before I felt like I didn't own any explanation to other people. If they ask why why we homeschool I say that DS7 is 4 grades ahead in math and DS5 is reading Harry Potter
I say it as a matter of fact and it usually takes care of it. I very often add that the kids attend a private school 2 afternoons/week which takes care of the possible questions about socialization
DS7 went to private K and I think people gave me much harder time then than when we started hs. Sending my child to a private school was taken as a clear sign of me not liking the public school where all the other kids went. The fact that we are homeschooling now is way too crazy to really bother most of the people
