I think that self-esteem comes from mastering challenges. Doing hard things well is one of the best ways to develop a sense of accomplishment and an internal locus of control. With that in mind, I think that there is no doubt that the public GT program is a better place than the girl's school for your daughter to meet the kinds of challenges that will build her self-esteem, not break it down. I think that the fact that the girls' school doesn't really offer advanced math is likely to send the message that math isn't something that girls are supposed to be good at - not a message that you want to send to your daughter at this age.
Edit: I would like to mention that I went to a co-ed public middle school, and nearly all of my best friends there were boys. A fair number of them are still very close friends today. I didn't really feel any pressure to "dumb down", but maybe I was oblivious to it. I think there is something to be said for learning to work cooperatively with both genders and stand up for yourself in a co-ed environment, unless your daughter plans to work in an all-female company when she gets out of college.
