I think just give him the opportunity to play with numbers in any way he likes. I certainly wouldn't drill math facts, but you may find if you introduce a sum game he clamours to play it - or not, and that's fine. My DS used to love specifying how many of something to eat he wanted (grapes, cherry tomatoes, peas...), and would announce how many more he needed, how many he'd eaten, how many he had left, etc... I remember drawing the line at counting out 97 peas or something some time.
Don't be surprised, by the way, if he has a long fallow phase in which he likes numbers for counting and maybe simple sums but isn't very keen on doing more with them. My DS, like yours, got numbers early, but then his attention was diverted onto teaching himself to read - a highly mathematical process, if you think about it - and for a couple of years I thought maybe he wasn't destined to be very mathy. Boy was I wrong - when he got interested again, he went off like a rocket (basing everything on a totally solid understanding of number that other children didn't seem to have). I'm glad I didn't push him to maintain the interest while it was lying low. Of course YMMV.
