Gifted Issues Discussion homepage Forums GT Research Ignore ACT Reading and Science scores?

A new paper finds that the Reading and Science sections of the ACT have no predictive validity for college grades -- only the English and Mathematics sections do. The ACT is used as a talent search test for 7th graders, and its "younger brother", the EXPLORE, which has the same four sections, is used for students in lower grades. Programs such as Davidson Young Scholars should study if a composite using only the English and Math scores is better for selection purposes.

http://faculty.chicagobooth.edu/devin.pope/research/pdf/Final%20AEJ%20Paper.pdf
Improving College Performance and Retention
the Easy Way: Unpacking the ACT Exam
Eric P. Bettinger
Stanford University and NBER
Brent Evans
Stanford University
Devin G. Pope
University of Chicago and NBER
May 2011
Abstract
Improving college performance and retention can be difficult. We propose a simple and low-cost change in the way colleges use the ACT exam in their admission decisions that can greatly increase their ability to identify students at a high risk of under-performing and dropping out. Specifically, we show that only two of the four sub tests of the ACT, English and Mathematics, can effectively predict outcomes in college. This result is robust across various samples, specifications and outcome measures. We demonstrate that by eliminating the noise associated with the two non-predictive sub tests, student-college matches can be significantly improved.

My dd got almost the same score on reading and english, so I guess that I don't need to worry about the reading part not being predictive for her. She usually gets a slightly higher score on English (EXPLORE, ACT) than reading anyway. In fact, I think that her English scores have always been a bit higher.

I find it interesting that the parts of the test that rely more on the ability to apply standard rules and algorithms correlate more highly with college success than the parts that require analysis and interpretation.

Of course, weaknesses in either mathematics or standard written English would have a disproportionate effect on college performance, as mathematics problems solving and writing are the two primary outputs used to evaluate mastery of the curriculum in college courses.

Along the lines of standardized tests, I've been learning about the GRE recently. Did anyone here know that it's impossible to get a GRE score that's three standard deviations from the mean (general and subject tests alike)? That is, except on math, where a perfect score of 800 is at the 94th percentile and therefore isn't even two SDs above the mean. Likewise, you can't go 3 SDs below the mean without going under the lowest possible score of 200.

This document is produced annually by ETS (the folks who write the tests). It shows average GRE scores and standard deviations. For the general Verbal test, the average is 462 and an SD is 119. The 3rd SD would be 819.

This information, which is from a published paper, discusses how many SDs you need to discriminate between strong and weak students (a lot more than two-ish.).

Do most graduate schools in the US rely on GRE scores? It seems to me that the test is terribly watered down and mostly a tool for measuring averageness. This is what we're looking for in graduate students? Averageness?

I don't get it. Well, I do but I don't want to, so I'm blocking.

Val

When I went to graduate school in one of the natural sciences, schools did require the GRE General and the relevant subject test http://www.ets.org/gre/subject/about . I don't think that has changed.

However, the 94th percentile on the GRE isn't the same as the 94th percentile in the general population or even as compared to undergrad students b/c it is a select group that takes the GRE. That group is, presumably, well above average in terms of achievement at least in that they are planning to obtain at least a Masters degree. Being in the top 6% or higher amongst that group is likely indicative of pretty high achievement.

But the point of the test isn't to measure achievement in comparison to the general population. The GRE is supposed to measure achievement among people who are high achievers already: people who did well enough in college to continue on to grad school.

The math section has nothing harder than basic high school geometry: there are no questions about statistics, no trigonometry, and no calculus. Most of the questions are straightforward, with only a few slightly tricky ones.

My feeling is that the GRE fails as a measure of talent or achievement among people who are supposed to be talented high-achievers. It measures averageness. And the quantitative section measures this on junior high and high school math, no less. Verbal is not much better, and the subject tests don't discriminate well either.

If you look at the average scores of people who major in education... well, the numbers are pretty depressing. The results are saying that people who want to become teachers can't do basic algebra and geometry questions, after supposedly having studied these subjects for this test.

Overall, I don't think the scores reported by ETS say much that's good about the aptitude and/or study habits of the majority of test-takers. Plus, even if the super-smart super-high-achievers can only score inside 2 or 3 SDs of the mean, what does that say about the level of cynicism among the people who wrote the test?

I'd agree in regard to the math level of graduate education students but based on a limited sample. I got my Masters in Public Health and took a few graduate classes in the Statistics & Research Methods dept with people who were getting Masters degrees in Education.

When I took the GRE it had three sections: math, verbal, and analytical. I got a nearly perfect score on analytical, close to that on verbal and high on math but not near perfect. In other words, I'm above average in math but not fantastic at it. I clearly recall one of the stats classes where all of the people in the class save for me and one other person couldn't seem to understand the concept of doing the parts of the equation that were within the parenthesis first (PEMDAS -- order of operations). I and the other person who got it spent nearly the entire period going around helping the other students figure out where they needed to start on some rather simple equations.

These people were nearly all Masters in Education students.

Well, you're above average at GRE math! The GRE doesn't measure your ability to find creative new solutions to math problems by thinking slowly and carefully (my other major complaint about the GRE and other standardized tests). It only measures how fast you can crunch a standard set of basic calculations. I understand that this skill is important for some things, but it's not all-important by a long shot.

I found an old document from 1996 that shows that score percentiles have shifted only slightly since then. It says an 800 was the 98th percentile but also says that +2 SD wasn't possible on the test, so interpret as you please there.

Again, even 15 years ago, education majors still scored at the bottom. They had the lowest scores for analytical and quantitative, and were only 3 points higher than the lowest score in verbal (business students scored the lowest there). Not too impressive.

With respect to GRE math, most graduate programs in mathematics require the GRE subject test in math, which is much more challenging and covers all of the material one would go through for a BS in math (calculus, topology, real analysis, set theory...). I'm not sure what other subject exams are given, but it might be that the GRE general exam is used to show competence, and that the subject exam is to differentiate ability.

I'm also a bit dismayed that the education students score so low and are assessing children on abilities in these areas...