I also have felt like this with DD when she was little and now with DS. The content he chooses to read is getting more out there. I thought I was over this but the other day I had to leave him for a second to take DD3 to her teacher across the room and he went over and started talking to a man about the physics article he was reading. When I came back the man was very shocked and commented on how smart he was. I said thank you and moved down but then we had to sit there and wait for 30 min with DS5 asking me questions. I tried to change the subject but it didn't work. It was pretty nerve racking.
The next day DS met a woman from Africa while I took DD to her teacher. When I came back he was showing her his field guide on bugs and the woman had seen many of the bugs and knew a lot about them. DS spent 45 min talking to her about the bugs and had a great conversation. She also commented that he was smart but it was okay the way she said it. She had a conversation with him and was genuinely interested in talking to him and was not treating him like a circus bear doing tricks so it was okay.
I think it depends on the intent of the person commenting as to how uncomfortable the comments are. If people are genuinely interested in meeting another like minded human being and having a conversation than it is fine. I think most people do not like to feel different. Meeting more families with HG to PG children has helped me be more comfortable with the comments. We also did relocate to an area that is more friendly towards our kids. It is good to hear that other people struggle with this too.
