Another great source of challenge that may keep you a little closer to grade level for a while is math contest questions. Math Kangaroo seems to be particularly "non-curriculum-y", with more problem solving, logic and visual puzzles. Different kids love different kinds of math, so you can experiment with the many fantastic books above to see what appeals. Numberphile videos on Youtube are also wonderful to view and discuss together.
SaturnFan - if your son still has any lingering love for graphing equations, mine loses himself for days on the Desmos graphing calculator (
https://www.desmos.com). In addition to endless making your own to see what they look like, there's also a whole library of funky graphs and pictures people have made. DS loves to try and deconstruct them and understand why the equations used produce the results seen.
