I felt this way when my children first started at our gifted school. Over the years, I've come to realize that most people can be easily distracted if you keep the sentence going with some other pertinent information.
For example:
Person: Where does your dd go to school?
Me: She goes to Blank School. It's near the intersection of Main and Elm about 20 minutes from here. So far the drive is going well, but we'll see come winter!
or
Me: She goes to Blank School. It's quite a haul, but I have a carpool which really helps. DD really wishes she could take the bus. Does your dc like the bus?
Then we're off and talking about the bus or the drive rather than the reason dd goes to the school she does. I swear, it works 95% of the time.
Plus, after a few years, I think I'm less sensitive about it. Either they know the school or they don't. If I have to explain it, I'll often say something like, "DD has some special learning needs that make this classroom a better fit." People rarely push after that.