Obviously students at rare levels will find it hard to find a large local cohort, by definition. There's no getting around that, and so they may need some individual attention, some resources for self-directed study, connection to peers in a larger are, etcetera.
But what about larger groups at the upper levels. What about top 5%, top 10%, top 20%, top 50%. Or what about students between percentiles 75 and 90? Why shouldn't all students at all levels be entitled to be taught at their approximate level? There should be classes catering to all levels, and placement in these classes should be purely meritocratic. This merit should not be compromised to fulfil other types of quotas. As long as everyone is being taught at their approximate level, how could it possibly matter if there are measurable demographic differences between the classes? I've never seen any convincing argument not to do what I'm suggesting.
When people don't like the situations their kids are put in in schools, they leave if they can (or don't enter in the first place).
In our district, the gifted programs are generally in lower performing schools, schools with more discipline problems, schools in more dangerous parts of town, and so on. There are various reasons things are done this way. But a consequence is that a lot of gifted students don't participate in the public school system.
The gifted classrooms themselves are meritocratic, and they are an incentive for qualifying students to participate in the public school system, but when you consider how bad the schools are that contain the gifted, many students and parents say no thanks. The gifted community would love to have a single K-12 dedicated gifted school in a safe area, but the district has no purpose for such a school.
Schools and districts use gifted (and above average) students as a commodity that can be placed and moved around as pawns in a sytem, basically to manipulate demographic and school score statistics. This is so unacceptable to large numbers of students and parents that many are effectively left disenfranchised with no acceptable public school option, and are pushed into options that many can hardly afford.
