Hm, I always found it more awkward when the stranger doing the commenting didn't have a kid like that and wanted to know how I taught him to read at 2.5, or would offer up some other unsolicited and strange comment/question.
When I asked how to deal with that a few years back I got some good advice, which was to say something like, "oh, he just loves books", or "he's always been into numbers". It acknowledges their question without really answering it and also subtly implies that you didn't drill with flashcards.
Now, as far as a secret handshakes and such, I just start out with something like, "your son reminds me so much of mine at that age," or "oh, my son is really into space too -- they would really hit it off", just naturally commenting on whatever it is that is making you think they're gifted. Why does it have to be harder than it is for parents with kids in competitive swim team? It's unlikely that anyone is going to be offended that you have noticed their child is smart.
I remember having a great sense of dread when DS had just started to read and a friend with a child the same age asked me if she had really just seen him read. She said, "tell me the truth." (gulp). But then she said, "my brother did the same thing." And I was so relieved I almost started to cry. I hadn't been able to talk to anyone about what he was doing.
Over the years I've found that other parents who've gone through school or behavior problems or whatever already are very willing to share and be helpful, because they also have experienced the isolation and overwhelming responsibility of blazing a trail for their children.
Good luck!
