My son had the opportunity to attend our district's gifted school from 2nd - 5th grade. We chose to keep him at the school he was in in Kindergarten/1st, for several reasons. He did just fine, they accelerated him a year in math from 2nd grade on, and did a great job of differentiating in other areas. He is now in the middle school gifted program (6th grade), which was brand new this year, and at the school he would have attended anyway. Most of the kids in the program did go to the elementary gifted school, but that doesn't seem to bother him, he gets along with pretty much everyone. All in all, we're glad we kept him where he was - it's a Title 1 school, so a low income area - but incredibly diverse, and I think it's been great for him.
