Your point about the multiplication facts is very valid and the amount of time it would take him to compensate for not being fluent in those facts.
You've already received great advice above and I don't have anything to add to it other than, it's important to know and understand the math curriculum and program in your own elementary school and school district. For instance, in our schools, fluency in math facts wouldn't really be a barrier to getting work done until a student reaches pre-algebra, and once students reach pre-algebra, students are allowed to use calculators. My dysgraphic ds was never able to do any math fluency worksheet quickly but he had no issues with multiple grade math acceleration. Ability to understand and manipulate fractions, however, was something that slowed down quite a few of my dd's friends and was an obstacle once they hit pre-algebra/algebra. My gut feeling on that was it was due to the our elementary school math curriculum in the early grades, and my dd didn't have the same issue because she'd had a traditional math tutor take her through fractions etc (she'd fallen behind in math due to vision issues and needed outside tutoring to make up for lost time).
Best wishes - I hope you're able to work out a schedule with the school!
polarbear