Connie, I hate to say this, but I think it is very likely that your son's classroom wasn't disproportionate by chance.  If it was 50% ESL, and the school has a much smaller percentage of ESL students, it was quite likely deliberate.  At any rate, I think that your experience illustrates the problem of having special ESL classrooms.  It does cause segregation.  It does keep ESL (and sometimes fluently-English-speaking minority) students from getting the education they need.  It is a form of institutionalized racism that keeps ESL and minority kids from having access to an appropriate education.
Regardless, from what you've written elsewhere, I think it is pretty clear that your son needs differentiated curriculum.  Even if all the ESL students were removed from his classroom, he would not be a normal child in that classroom.  He would not be a normal child in a gifted classroom, if he is working 6 grade levels ahead in math.