Have you had him tested and/or do you know where he is at ability & knowledge wise in math? We were pleasantly surprised when the gifted coordinator at our DS's school gave DS the end-of-year assessments for 3rd and 4th grade math, which has resulted in an additional acceleration into 4th grade math. (He had managed to get through 2nd grade math basically on his own as a 1st grader last year, then we supplemented with some online games and voila, that was enough.)
I would keep listening but be thinking about things like the Iowa acceleration scale as a tool. Have you read "Developing Math Talent?" It was suggested to us by our son's tester and although I am not through it yet, I am finding it very helpful and practical.
