We also didn't force memorisation of facts before moving on. I feel like memorisation and rapid fire recall is a separate skill which is more age-based, and forcing it would have taken all the fun out of it. So for four years one of my kids progressed through math knowing how to multiply and divide but not knowing the times tables, then BAM, third grade hit and it was all memorising Pi and memorising this and that and everything else. Sure, my five year olds can do third grade math, but they're not third graders.
We used one of those grids for help, especially with division. Have you tried Kahn academy? It's pretty fun and they can follow their interest.