Nothing wrong with that.
Though I think I should note that many kids have intellectual "growth spurts," where they make huge leaps in one area all of a sudden. This happened with both of my boys in math at different times. DS4 was counting at Christmas, but nothing more advanced. Then WHAM! By the end of January, he's suddenly rivaling DS7 in arithmetic! He does things in his head that amaze me! And it was just all there one day. Like he just got it all of a sudden. Growth spurt!
DS7 had a similar "growth spurt" in math this year, though not quite as huge and astonishing. But he was just ripe for math, and he has enjoyed learning it. I don't think his reading has advanced a whole lot this year, but his math leapt ahead!
Don't write off the math. Keep using it where it fits in playtime. Avoid the worksheets if he's not into them. No reason to push him.
For multiplication, blocks are good. Make arrays (squares and rectangles) and race him to know how many blocks are there the fastest. Talk about how he has to count the blocks, while you can use math to find the answer. He'll want to learn how to do what you're doing.
Cookies and pizzas are good ways to do division. "If I have 24 cookies and 2 boys, how many cookies will each boy get?" "If our pizza has 8 slices, how many slices will each member of our family get."
That sort of thing.
