In our district most kids start being differentiated in 7th grade. At the start of Junior High.
The only exception is the GT class my son was in 4-6th grade worked one grade "up" in math starting in 4th. And in 6th the entire class were taking pre-algebra. This was the only class in the district doing this. The other GT classes and GATE students in the normal classes just got enrichment in math from 4-6th grade.
At the end of 6th every student in the district takes a placement test to see if they are ready for pre-algebra. If they pass the pre-algebra test, they are offered a chance to take the algebra readiness test. Students take one of 3 math classes in 7th, Honors Algebra, Pre-Algebra, and "7th grade" math depending on test scores. In the district there are one or two students every year or so that take Algebra in 5th or 6th grade. I am not sure how they petition to take this, but I have known a few of these students. One was in my son's class as he is taking Calculus as a freshman in H.S.
The largest percentage of kids take Pre-Algebra in 7th and Algebra in 8th grade. (60%) About 15% of my son's 7th grade class took Algebra in 7th grade and Geometry in 8th grade. If they weren't enrolled in my son's class they were never really enrolled in Pre-Algebra.