The program that you're describing sounds similar to the program my dd is in.
She is in an all GT 3rd grade classroom that occasionally does special Social Studies activities in which they mix with the 3rd grade general ed classes.
For my dd, it has been great. The accelerated academics at least help avoid the lag that she experienced in a general classroom while the teacher did review after review. In the GT classroom, they expect the kids to get it the first time and they just move on. They did not teach cursive or multiplication/division tables like they spend so long on in standard 3rd grade curriculum...they just expect that the kids can pick it up as they go along and I would guess that 95% of them do.
But most importantly, my dd seems to have really found her "people" this year, and that is more important than academics to me. Nothing compares to being surrounded by kids who think in a similar way. They all have a passion for learning and have their own little quirks and obsessions...so no one is singled out for being different.
In our county, the GT centers are also often in less than desirable neighborhoods and/or lower performing schools (perhaps to boost the school's overall test scores), but we are fortunate that our GT center is actually in an upper-middle class neighborhood.