Can you do more to integrate science and math and do more problem-solving? Having him use math in experiments and such or to solve problems will mean that he's getting math thinking without rushing through a linear arithmetic-only curriculum.
Keep in mind that arithmetic is not the sum total of math, though most of what kids usually get in an elementary math curriculum is simple arithmetic. That stuff an HG+ child can often whip through ridiculously fast. But mathematical thinking isn't necessarily a linear path and it is a lot more than just arithmetic.
I'd recommend Singapore Math's "Intensive Practice" workbooks to you. They're really challenging math thinking, though the arithmetic is not hard.
I worked on geometry with DS-then-6 last year. We used Lynette Long's "Painless Geometry," which is a simplified without being dumbed-down version of high school geometry (fewer proofs, more pictures...), and it was good for him. It was light on arithmetic and long on concepts. It was visual. It was challenging and yet not over his head.
And that's how I used geometry with a 6yo.
It wasn't in preparation for anything else. It wasn't on any linear curriculum path of any sort. If anything it was a total deviation from the usual math path! But it was math and it was good for him and it was fun. Does he remember everything he learned? No, I don't think so. I don't consider him to have passed high school geometry. But remember that curriculum tends to cycle through concepts. Kids don't usually get things once and never see it again. So whereas other kids his age were learning "this is a square" (which they'll get again at a higher level in 3rd or 4th grade, and then again in junior high, and so on...), DS-then-6 was already learning how to use a protractor and what angles are congruent in a given set of crossing lines. He needed concepts, so that's what geometry gave him.
So, I guess what I'm saying is that you're not doing anything wrong except perhaps that you're accepting a view of math as linear when it doesn't really have to be. And even that's only "wrong" insofar as it isn't really working for you and your son!
Try more problem solving, more real-world applications of math, more graphing on Excel or other software, more computer programming-type math...less focus on arithmetic. You can still do some, of course. But if he's mastering that, do something harder and more fun!
