And most parents won't teach their 5yos multiplication and division (other than as a one-off - I once demonstrated long division to DD-then-in-first as a tool for factoring, but she clearly wasn't ready for 4th or 5th grade work in general), even if the kids are capable and interested. Whereas they will let their kids read harder books without even thinking - I was shocked to see that the Rainbow Fairies books are 4th grade level in AR!
I think you've hit the nail on the head, particularly with a mathematician as opposed to an arithmetist (love that invented word!). Spontaneously inventing long division is rather unlikely, even for Terry Tao.