I would not wait until your DS has mastered math facts before moving on. DS8 still does not have his times tables memorized, but we decided to move on and work on them over time.
You might also look at the Singapore Math Challenge Word Problems books. The actual arithmetic is straightforward, but the methods needed for solving the problems requires some thought.
For LoF, we started with Apples and tore through them quickly. Yes, all of the early books were too easy, but DS liked the story, and he learned stuff (e.g. sets) that was not covered in public school. We're on hiatus after Fractions, but we've got the books through Algebra. DS likes to read the physics, biology, and econ books on his own.
