I've felt and done much the same as you. I've tried to keep things as quiet as possible and mention it only when either I'm talking with another parent of a gifted child who is specifically asking about our school experiences or very, very close friends who know I don't consider giftedness this 'my kid's so amazing' thing, but something that causes me concern at times. But when your kid is in a different class and that fact is relayed to kid's parents, as you say it's simply impossible to pretend it isn't real. My experience is that the chatter will probably die down after a while, although you may have a few disgruntled parents. That simply is the school's job to sort out whilst you continue to advocate for your child.
