I think that almost certainly plays into things. The fact is that most prodigies are educated in very unconventional settings.
That alone allows for specialization to proceed unimpeded by 'working on relative weaknesses' or 'discouraging over-reliance on relative strengths.'
Given what we saw happen with a PG child over just a two year homeschooling stretch-- which wasn't even strictly child-led, mind-- I think this is a very real possibility as an explanation for the development of prodigy-level ability. My daughter went from reading beginner chapter books to discussing adult literature quite competently, and was well on her way to doing likewise with science topics. Her reasoning and critical thinking skills were close to prodigy-level at that point (she was then just-turned 6 yo, and really was making fairly insightful observations re: poetry, military tacticians, historical events, experimental design, etc). On the flip side, though, she still could barely write even her own NAME legibly, much less communicate via a written and coherent paragraph.
Oh, there is no doubt that my DD wouldn't have become a prodigy at anything recognizable to the outside world (math, music, chess or the arts seem to be the most commonly recognized sorts of prodigy). Partly I think this is because she seems to lack that particular single-minded drive to mastery in those areas... and partly because she didn't have an area of strength in any of them which vastly outstripped her other areas of high ability.... but also because we didn't regard that kind of obsessive pursuit as healthy, and we actively discouraged it, with the help and advice of professional educators in the family.
When one looks at musical prodigies throughout history, they've mostly had that interest actively fostered, and they've been allowed to fixate on it (and in some cases, even encouraged/forced to do so).
One reason why prodigies are relative rare, I think, is that so many different factors must be present to allow it to occur. Extraordinary working memory is probably only one of them.
