You're not incoherent at all.
It sounds like others are noticing that your child is different, which actually can be helpful! Some parents here struggle with getting their child identified as gifted. If everyone you meet pegs your son as advanced, you may have nothing to worry about on that front.
I also wouldn't give up the ghost just yet on finding a good school fit for him. Moderately gifted kids can do well in a range of environments, even some highly gifted kids may do all right with little in the way of accommodations, and you don't know for sure yet how gifted he may be. Hopefully when he enters preschool next year, he will find another four-year-old at or near his wavelength. And the local schools may surprise you with what they're able and willing to do, if accommodations of some sort become necessary.
I would personally try to relax over this, focus less on school for now, and take the comments for what they are. I think a lot of people who make such statements must feel like they're simply complimenting your son and showing interest in your family. You can also steer the conversation away from topics that you don't want to talk about. You don't have to be confrontational with the other person; you could start talking playfully about whatever little-boy qualities you see in him, for instance. You could then use that discussion as a springboard to talk about the childish, fun aspects of your son.
