I would agree that it's not crazy, but it certainly hasn't been a factor at all in our decision-making. We had no clear expectation of children who were NT, MG, HG, 1e, 2e, or anything else when we chose a place of residence (though >= MG did seem reasonably likely, and I had a suspicion that another e might be in play, based on family history). We did, however, know what the range of services was in the general area--and that it would range from good to horrific for disabilities, and better-than-average to horrific for GT (stronger services for both exceptionalities being in the same school district). We also knew that the better property values were not going to line up with the better services, due to the inability of the general public to do accurate data analysis.
Which is why we began community selection with other criteria, always reserving private and homeschooling as options.
We happened to end up in a community with a strong educational reputation, so we also get to smile vaguely when others laud our public schools.