I don't have a lot of advice for you, but I do have a great recommendation. My DS6 fell in love with the Rightstart Math program last year, and it sounds perfect for your son! It's very manipulative-based, and the premise is to teach the logic/understanding behind math instead of just rote memorization like many standards-based school math programs.
For example, when learning about place values, they first explored the idea on an abacus, then used picture cards with lots of "trading" games (10 1s for 1 10, etc), then to place value number cards, then on to adding with carrying. The card games that are tied to the lessons are very fun to share together, and it's amazing how much mental math they quickly pick up.
RS A is designed for kindergarteners, but it's easy to move at your own pace. Mine is at the end of C now, and is still having fun. He learned his multiplication facts cold in about 3 days from the way it was taught here, mostly through games.
Oh, and btw, they say subtraction is much harder developmentally than addition and even multiplication. In RS, they move through multi-digit addition before they really even approach subtraction (in 2nd grade materials). As you move forward, maybe continue with the addition concepts before subtraction? LEGOS are also a great tool for boys to learn multiplication!
