I run into this regularly because people ask me why we're homeschooling. Because I really don't want them thinking we're religious conservatives--so not me!--my solution is to say, "School wasn't working well for him because, well..." and then lower my voice to a whisper to say, "...he's pretty bright."
Reactions to that approach have been universally positive/neutral. No backlash or dumb comments like the ones you've received, S-T. Apparently the whisper and the lack of use of the word "GT" conveys to people that I'm not bragging. But it gets the point across.
I realize it's a different situation because of the homeschooling. But maybe this gives you something to work with?
Also something to consider: are these real friends, or are are they acquaintances who say such things? Dumb comments tend to bother me a lot less from acquaintances than they do from friends who realy know me and know my kids. I, too, avoid talking about all things GT with people I don't know well. But my *real* friends get it and are supportive. Do I talk about it a lot with them? No. But I do have a few friends IRL--all with at least MG kids themselves who have had trouble with the schools--with whom I can discuss the problems we've had with school and possible solutions.
Being a homeschooler helps me find those people though, I think. It's like I have a neon sign that says "Different and okay with it" so people aren't afraid to approach me to ask questions. Just this week, a woman with a DD who sounds HG+ stopped me at pre-K to ask me about homeschooling. I suggested DYS to her and invited her to our local GT parent group meeting. Now we're talking playdate and maybe a family Dungeons and Dragons game.
Those people seem to seek me out since I'm pretty visible. I suspect the Judgey McNasty types stay away from me for the same reason.
<shrug>
