I wouldn't want him to miss recess. Social life is so important at that age! There are plenty of other ways to learn math. My 10-year-old has never learned math in school (used to do Kahn Academy at home instead of math homework; now going to an Art of Problem Solving class on the weekend). Usually in elementary school there isn't so much homework that you don't have 20 minutes for learning math in the evening. Plus it sounds like your school is pretty flexible. Maybe they would let him work independently during math time?
But if he really wanted to give up recess for the math class, I would let him--with only one or two gentle attempts to dissuade him.
