It's hard to know when to take a kid out of a school and try something else, because there are no crystal balls. So much depends on the teacher and if you are lucky enough to get a good one. I am not shy anymore about telling principals that my child has X issues and needs Y in a teacher. But sometimes none of the teachers in a particular grade are going to be a good fit (or the principal has an inaccurate view of the teacher, thinking they are better than they really are). Of course, by middle school the child usually has more than one teacher and there is a higher probability of encountering at least one incompetent fool. Then the administration makes a big difference, and how much they understand your child and the issue, and want to help.

I think that once you notice a child spiraling downhill, it's time to seriously consider other options. For instance, the child is showing anxiety, thinks negatively about themselves, is starting to hate certain subjects or think they are bad at those subjects, etc. If advocacy efforts don't work, then that leaves little choice. I have been criticized for school switching, but the criticisms come from people with perfectly ordinary children who have no concept of what we, or our kids, face. I think it's almost always better to switch (or homeschool) than to keep a child in an environment that's toxic. Keep in mind that he may claim to be happy (because switching schools is scary and going to the unknown), but you need to look beyond that at his actual behaviors and what's underneath. If there are no better options, then you are left with no choice but to advocate the best that you can.