I do think that the early signs of giftedness and spectrumy (new word btw) issues look similar. And then there is the overlap. I think the difference between quirky giftedness and ASD traits can be so similar that you may not be able to tease it out at 4.
It's nice that you can relate to his early math potential. I was the kid constantly lost in my imagination and creativity who had no need for science or math unless it served to further the plot of my current play/story/poem. My ds' obsession with the fact -based world totally freaked me out.
I realized that there are different ways to be kid.
