DS is having a surprisingly good experience right now (first grade) under somewhat different conditions. A new child recently joined the class without a word of English - he now has an aide but for the first few days the teacher was using Google Translate. DS clicked with him so the teacher paired them up. Now they are both given extra leniency in moving around and talking, DS is stretched by learning a bit of Spanish, and the teacher is also giving him extra privileges (like iPad time to pursue his own interests) as a reward.
But... I do remember feeling a lot of anxiety, at this age, about working with other kids and what I should do if they didn't understand. I think you are right to make sure there are limits set and your DS is learning at his own level, not just patiently working with agemates.
