I WISH I had people respond, "Wow he's advanced for his age." I'd be much more comfortable giving his age if that was the case. What I have gotten, for the most part, is dropped jaws and "A 4 year old that does ____ !!!!!"
One thing we talked about with our child is that strangers generally mean well when they say stuff like this. People will NOTICE difference. But noticing is not the same as criticizing and they don't realize that you've heard it many times before. I find this applies to kids who are tall, have pretty eyes, etc. People are just making conversation and it is best not to worry too much about that. And, I'd keep in mind this is very much a temporary thing as far as the degree of scrutiny. It tends to fade as they get older.