I think it's definitely something to keep your eye on, and if you are worried about autism, you should get him evaluated by a developmental pediatrician. My guy was obsessed with numbers and letters at a really young age. We just bought him math manipulatives, and let him watch YouTube videos about numbers. He was diagnosed with autism when he was 5, and he gets ABA therapy 4 days a week at home. If you met him, you would just think he's shy. He gets speech therapy at school, it's just once a week. For me, the main reason I got him evaluated was because he was so asynchronous. Just so advanced in math and reading, and so behind in verbal communication and social skills. Now that he's 7, he's not super obsessed with numbers anymore. He has friends, and loves being around people, but still has a hard time with verbal communication, but is being much better from therapy. I would say just trust your gut, and if you think there are issues, have a specialist evaluate him.
