I think it comes down to the stats of the kids who apply to the elite schools. Most applicants have stellar GPAs and SAT scores; the admissions folks need to look at other aspects of the application. Somewhere I recall reading that about 85 percent of Yale's applicants are "qualified", but since they admit less than 10 percent, other factors come into play.

For example, here are some stats on Brown applicants/admitted students:
http://www.brown.edu/admission/undergraduate/about/admission-facts

You have 2,457 applicants with an 800 SAT Math score. Brown admitted 2,759 students, so if they went strictly by scores, then you probably shouldn't even bother applying if you were less than perfect on the math section.

A kid with a 770 or 780 in math (one question wrong) isn't a weaker applicant than the 800 kid. Once you get to a score of about 2250 on the SAT, there isn't much difference from a perfect 2400 (just a few questions wrong on each portion can bring you down 150 points).