This also gets at the fact that IQ is closely linked to the family's income level and the mother's (and probably father's) educational level. IQ isn't simply a set-in-stone number that cannot be changed up or down.
Studies have shown that babies in wealthy families are exposed to more words and more complex words during development than babies in poor families. Maybe that alone would affect your verbal IQ tested at age 5.