I am not sure about the advisability of avoiding some things, but I understand when you are feeling unsure of how to approach a subject. The only thing that came to mind were the very cool books on math that are more about sitting down and reading a book about infinity, square numbers, etc. and getting into the fun of math with out 'workbooks'. A couple I can think of are 'The Number Devil', 'Penrose the Mathematical Cat'.
If he's not into reading them himself, the Number Devil is actually a fun one for a bedtime story.
(My ds is not pg, he just enjoyed hearing about the story and the ideas about numbers, maybe around 6 or so)
