Unfortunately, it does matter.
Many/most prestigious or competitive graduate programs take people from their undergrad programs. You would think it is purely a meritocracy but it's not.
As an example- I graduated from Harvard Medical School, having gone to Northwestern undergrad (currently ranked #12). I was the first person to go to HMS from NU in 20 years; NU hasn't been a feeder school for HMS. I would estimate that only 10% of my HMS class came from non-IVY league, non-top 20 schools. At least half came from Harvard, Yale, Princeton, etc. undergrad.
It's easier to get into a prestigious undergrad from high school than into a prestigious grad program not coming from a prestigious undergrad.
Obviously, it can be done. People do it all the time.
And there is a separate discussion about if you need to go to an expensive, prestigious school for grad or undergrad school at all.
My point is simply that- you may (or may not) still be able to get into a more prestigious, expensive grad school coming from a no-name undergrad. It is very hard to do.