Well, asynchronous development is real. It sounds like this was just a bad experience following a good one, so you would have no way to predict your kiddo's behavior. But for my DS, we didn't put him in classes that were for older kids until he was 6, since he was a frustrated outburst kind of kid. Well, at least at home; as it turned out he's a well-behaved teacher pleaser kind of kid at school.
There was another developmental reason we didn't put him in classes for older kids. For some of the fun science classes where part of it is dismantling things to see how they work, DS would not have had the hand strength for the tools they were using, and it would have been frustrating for him and for the teacher who would have had to do it for him.
But I hear your pain. We have had similar "he is young" conversations at schools where it had nothing to do with his behavior, but rather about what the staff thought he should be learning at that age (not the same as what we had asked for).
