I'm convinced that many people have savant skills that they either learn not to display to others, or simply never have much occasion to demonstrate.
Or they are things that aren't especially flashy, but also not exactly alluring, either.
Memorization of numerical or spatial data is often in the 'flashy' category with kids. But I know a number of adults who remember every phone number they've ever heard-- even once-- and can rattle them off in an amazing way. I have even known people who had that ability with auditory verbal information, and could "replay" conversations even from years before. I can remember color that way, and apparently have better color discrimination than 99% of humans (evidently).
It really is an inborn difference, I think. But I also don't think that these savant skills are all that rare. It's just stuff that doesn't COME UP naturally as children get older-- and almost never other than as, well, party tricks.
