Part of the stigma with the "gifted" label is the elitism that pull out programs sometimes create (I first read this in Genius Denied). In order to placate the rich white parents, district pullout programs might have kids go see a play or a field trip or other activities that don't specifically address a particular child's area of "giftedness". At one school in my town, parents have their kids privately tested again and again to qualify because the gifted program is almost a social stratum.
The charter school where 2 of my kids attend does not believe in labeling and all kids get to go to the theater, field trips, poetry workshops, etc. Everyone benefits from these enrichment opportunities. It doesn't mean that my son shouldn't go to a different math class (which he does). In a small school setting, I like including everyone in the special activities and not using the label with subject acceleration/deceleration that relates to "need".