I thoroughly recommend "What's going on in there? How the brain and mind develop in the first five years of life" by Lise Eliot:

It's a long time since I read it so this is based on a quick flick and possibly fallible memory, but I think it's true to say:
- they have almost all the neurons they'll ever have, at birth;
- but lots of connections (synapses, axons, dendrites) are made in early childhood (mostly before 2? I think your doctor is approximately right on this)
- in later childhood, most brain development is actually getting rid of unused synapses (though there's plenty of that early on too: lovely quote from p32 "Children lose on the order of 20 billion synapses per day between early childhood and adolescence."!
- myelination is crucial, and also happens fastest before 2.

Here's a quote from p187:
"Brains are composed of a great deal of fat, some 60 percent. Nervous tissue is second only to fat tissue itself in lipid concentrations. [...] Specific lipid molecules are needed to produce the cell membranes that cover the millions of miles of axons and dendrites in the brain. (Since neurons are so long and skinny, they have an exceptionally large amount of outer membrane compared with other types of cells.) The need for new membranes is largely confined to the brain growth spurt of infancy, when neurons are most busy spouting new dendrites and forming and reconfiguring their synaptic connections.

In addition, fats are essential to the process of myelination. Myelin is a dense material, composed of 30 percent protein and 70 percent lipid [smart alec comment: elsewhere she cites 80/20 not 70/30, presumably it's actually somewhere in between!], which together form a thick greasy coating that prevents water-soluble ions from leaking through the membranes of nerve cell axons (see Chapter 2). All of this massive myelination that takes place during the first two years of life requires several specific types of lipids."

She goes on to write about essential fatty acids, the role of cholesterol, etc.

So I think allowing for gross simplification, your doctor was roughly correct. I don't know what accounts for the increase in brain size from 2 to 5, but my guess would be "water"!

By the way, relevant to the thread discussing things correlated with IQ, old wives' tales or otherwise, she reports a modest correlation between head size and IQ: correlation coefficient of only 0.35, though, so "head size is of little value in predicting the IQ of any particular individual".
Email: my username, followed by 2, at google's mail