Unfortunately, it does matter.
  Many/most prestigious or competitive graduate programs take people from their undergrad programs.  You would think it is purely a meritocracy but it's not.
 As an example- I graduated from Harvard Medical School, having gone to Northwestern undergrad (currently ranked #12).  I was the first person to go to HMS from NU in 20 years; NU hasn't been a feeder school for HMS.  I would estimate that only 10% of my HMS class came from non-IVY league, non-top 20 schools.  At least half came from Harvard, Yale, Princeton, etc. undergrad.
   It's easier to get into a prestigious undergrad from high school than into a prestigious grad program not coming from a prestigious undergrad.
  Obviously, it can be done.  People do it all the time.
  And there is a separate discussion about if you need to go to an expensive, prestigious school for grad or undergrad school at all.
  My point is simply that- you may (or may not) still be able to get into a more prestigious, expensive grad school coming from a no-name undergrad.  It is very hard to do.