I think alot of this stems from parents and society saying that "I'm no good at math, I can't help my kid do their math and they're only in 3rd grade" and other such things while their kid is sitting there listening. I think that while there are obvious problems in the way our education system works, the larger problem is that there are so many adults who seem to be proud of the fact that they can't do math well. Being a math/science teacher I hear it all the time from parents and random adults I meet when they find out what I teach.
It is annoying and I always say that they should change their attitude or their child will form preconceived notions at their ability to perform well at math.