The timed drills are critical for later success in math and the other quantitative classes up through calculus.
I think we'll have to agree to disagree on that one. I had to count on my fingers and/or add-on for a handful of addition / subtraction / multiplication facts until I was out of college, and did fine through Calc 2. But the only things I considered to be "hard parts" were trig functions; the rest of math all made sense.
I personally found that as math got more advanced and complex, the ability to understand the underlying logic of the problem gave more of a speed advantage than being able to quickly recall facts.