Boy, this is coming my way. Believe it or not, I just saw this happen with MIDDLE SCHOOL admission. The kids at my DD's school, which is really a very challenging program compared to a typical elementary, did not do very well in terms of admission to our application-only middle schools because they were compared to kids at standard schools, and there was no weighting for the gifted program. The same thing will happen again when DD applies to application-only high schools. Having learned this did actually play into our thinking about which middle school DD should go to. Middle school. (It was just one factor, and a small one. But.)
To a lareg extent, though, I think the question is: what IS the challenge on offer? Is it something really good--really worth doing--or is it over the top with busywork, a teacher who never gives A's no matter what, etc.?
Then also, you have to think about the child's mental health...