I am starting a new thread on this topic following the topic having come up on a number of threads including this one: http://giftedissues.davidsongifted.org/BB/ubbthreads.php/topics/157401.html#Post157401

A few posters here have mentioned actual studies that show the CogAT to be a poor predictor of giftedness, but no one has posted any actual studies to my knowledge. The only one I had seen was the old one from the 1980s that I posted in the above referenced thread on the OLSAT vs. the WISC-R.

I have found a few others though, although they are not available in their entirety online as far as I can tell. Here's what I've got thus far:

An Examination of the Relationship Between the WISC-III and the Cognitive Abilities Test for Selection of Students for a Gifted Program - this appears to be an 84 page dissertation with the following excerpt/abstract:

http://books.google.com/books/about/An_Examination_of_the_Relationship_Betwe.html?id=mX5cOAAACAAJ

The Woodcock-Johnson III and the Cognitive Abilities Test (Form 6): A concurrent validity study - pdf available in its entirety online here: http://faculty.education.uiowa.edu/docs/dlohman/CogAT_WJIII_final_2col-2r.pdf

with the following abstract,

Thank you for doing this!

I may be the only one contributing to this thread - lol! Feel free to add, though, if you've got anything.

Here are some more studies, though:

A Comparison of WISC-III and OLSAT-6 for the Identification of Gifted Students (http://cjs.sagepub.com/content/11/2/120.short)


Identifying Academically Gifted English-Language Learners Using Nonverbal Tests: A Comparison of the Raven, NNAT, and CogAT (http://gcq.sagepub.com/content/52/4/275.short) - this may not apply to many of our kids, but is interesting none the less.

I have nothing. But I have to get my ducks in a row for dd6. I anticipate this being an issue.

To be fair, I do believe that there are a number of studies that show that the verbal and quantitative scales on the CogAT are good predictors of achievement in those domains and the nonverbal is less of a predictor of achievement. I'll edit to post links to some later.

Here's my hypothesis, though: I believe that the verbal and quantitative measures on the CogAT (the group ability test with which I am most familiar, b/c it is used extensively locally to me), are measuring exposure, knowledge, and achievement in those domains rather than ability in those domains. The non-verbal part, OTOH, which tends to be less considered by schools in determining giftedness, I believe measures visual spatial and abstract reasoning abilities more than achievement and also correlates less with achievement b/c it is more indicative of a divergent thinker not a child who can learn and regurgitate standard material easily. I'd tend to think that the non-verbal part of the CogAT is a better indication of giftedness as I define it which includes ability to create new knowledge and see connections and creative solutions more than repeat back knowledge that someone else has created.

Not really a study, but an author who tends to agree:

Increasing Minority Children’s Participation in Gifted Classes Using the NNAT: A Response to Lohman (http://gcq.sagepub.com/content/49/1/29.short) - keep in mind that this guy is the author of the NNAT as Lohman, who wrote the original critique of the NNAT as compared to the CogAT (above) is the author of the CogAT.
emphasis mine

This gets interesting here b/c Lohman, the author of the CogAT, has written extensively on his belief that ability and achievement are not separate constructs. I have to admit that I tend to disagree in that I've seen high ability people who are not high achievers and the reverse. I wonder if his thought process leads toward beliefs more in line with the current "talent development" model that the NAGC, for instance, embraces which requires high achievement and motivation to be considered "gifted."

For instance, here's a newsletter on the CogAT in which Lohman writes,


and

http://155.44.225.10/products/group/cogat6/pdfs/newsletters/CS_vol4_summer05.pdf

I do agree with him in some areas, for instance,

however I am not ready to say that we just call high achievement and high intelligence the same thing.

Thoughts?

eta: to drag another person into this, I was at the Mensa World Gathering years ago and attending a talk by Deborah Ruf about interpreting IQ tests. One of the audience members asked her about improving one's intelligence or IQ through practice or training. Her response (I'm not quoting, just paraphrasing what I recall) was that you can increase your IQ test score by practicing the tasks on the test, but not your actual IQ or intelligence.

These seem to be totally divergent beliefs on intelligence, no?

Cricket! Thanks! Very good information. Your quotes/research does tease out the achievement/intelligence paradigm of these instruments, and those who utilize them.

I think, in general, these instruments are very sensitive to environment and ability of person administrating. I wish that were better teased out!

In NYC they use the OLSAT, or did, for gifted programs because it is easy and cheap to administer and fair for those who cannot pay for IQ tests. I was astounded that it showed less gifted in the group in the Canadian study since so many kids score above 97th percentile in NYC.

I disagree with Ruf's view. They have shown that IQ points increase with string instrument study. They got 6-7 IQ points from one to two years of string instrument study and it was sustained and increased math scores.

Now you can be born with advantage, like being a poorer athlete than Joe next door, but you take Parissi training, improve your technique and it stays with you. You become a better athlete than Joe next door. You can say you were not born with the natural DNA or maybe you just weren't using it properly. Does it make a difference if the scores improve and sustain?

I believe, and hope that she'll forgive me if I'm misinterpreting, that Dr Ruf's thought was that you could improve the IQ score, perhaps even in a sustained manner with continued practice, but you would not be changing your actual functioning in day to day life in terms of your intellect and ability to do things that require that intellect.

One thing that I would like to see more of in these studies is a greater breakdown of the correlation between measures for various groups. I.e. - the overall correlation was x, but what was it for subgroups and, in those that so have that, how many false negatives were there vs false positives.

Wow what a treasure trove of research you've shared with us this morning!! I'm excited to wake up to all the work you have done here.

Thanks so much for opening this discussion. So many gifted programs strictly use the CogAt and refuse outside testing that we really do need a developed body of research to aid when scores vary so widely.

I get that schools want cheap testing and need to eliminate push parents who want little Suzy in whether she needs it or not... but yes, what about false positives?? And what of the Canadian study where HALF of the qualified applicants were not identified?

I also wonder if there is a predictable group or set of factors which correlates with being misidentified on CogAt. If THAT research were available it could allow schools to continue with their cheap group testing while offering the option for other testing to be used in that cohort only.

Makes we want to start a grad program in psych testing and start doing that research!

I don't really believe in "false positives."

I do think that what you get is a snapshot at the time. I think time of day , child's mood, child's feelings about the tester, the tester's competence all figure into how a child proceeds. I don't understand how a child could falsely score high. It would be sheer luck and guessing and fooling the tester (I am talking about individually administered tests). I do think that you can "improve" your score with knowledge of the test. I really don't think how you respond to a new structure of assessment should be a part of the test score.

My child took the SCAT with no preparation, and didn't get the score cut-off the 1st time. The second time (re-test), we went over the structure of the exam, and I answered a few questions, and he did practice questions. The next SCAT, his scores were good enough for the cut offs at 2 grade levels above the already above grade level test.

I guess that what I'm getting at in regard to false positives are kids who score high on the group test but not an individual IQ test. I'm not necessarily implying that their group test scores were falsely high on the group test but more that they are not necessarily indicative of intellectual giftedness but rather high achiever, convergent thinker -ness :-) ( sorry for the made up suffix! )

For instance, and this is a very small anecdote, but I know two kids locally who had high CogAT and/or OLSAT scores but average scores on the WISC with no unusual scatter on the WISC indicating depressed scores from a 2e issue or something else. They both also had high working memory and processing speed on the WISC relative to their other WISC indices leaving me wondering if one can appear gifted on a group test through a combo of high speed, memory, and in the box thinking, none if which would necessarily say "gifted" to me.

Yes! This is the type of "gifted" some programs are seeking really. The well behaved model student with motivation, speed, and memory. We walked away from a "gifted" school which clearly sought that type. They are the easy ones to dress up pretty and show off rather than the quirky EG, PG and 2e kiddos.

This leads to a thought I've had floating around recently particularly as my readings have been on very memory focused subjects. Basically is there a range of gifted that is reasonably successful, but overlooked and undernourished because their outlier skill is in long term memory?

Even to go as far as eidetic memory without a strong abstract reasoning/g level. It seems they would work well with some special differentiation that plays to their strengths. They may be even poorer advocates for themselves than the g-loaded gifted. Where the abstract reasoner is building out vocabulary rapidly from context, the memory talented might do better with early dictionary training. shrug

Interesting. This is what I've been thinking, especially after meeting with the principal regarding ds. She started off saying they have large gifted population (public school- and everyone wants their kid in gt program). Yet, when we showed her ds Wisc scores; she kept saying, "wow. These are very high scores." Dh and I found this funny since tester told us score was brought down by 2e issues and would be higher if retested later. Also, if the gifted population at school is so high, ds scores wouldn't be so shocking.

I just hope the gt will appropriately challenge and enjoy my ds. Really, I just hope ds has a better year at school next year....

Yeah, that's the spot we're in too. The large majority of the kiddos in our GT programming are the high achiever, in the box kids who can spit back out rote learned information very well, read fast, have their hands in the air a lot, teacher pleasers, A student types, but aren't kids who see abstractions, create new ways to do things, etc. My kids have had a harder fit and so many teacher simply misunderstand what gifted looks like IMHO.

That's where I have a hard time with Lohman's, the CogAT author, assertions that achievement and ability are essentially one and the same (and with the NAGC seeming to move in that direction too). It isn't that I am saying that high achievement means that a child isn't gifted, just that it isn't the defining factor in determining giftedness nor is achievement alone, even if very high, (in the absence of an IQ score in the top 2% or so) necessarily enough to support a gifted id.

We had the same experience with our dd14 when we brought her WISC-IV & WJ scores to the school when she was seven. The district GT coordinator told me to homeschool her b/c they wouldn't be able to meet her needs and one of the school admin people we met with looked at the report and said something like, "wow, this really is a gifted child!" It left me totally confused for the same reason as you: dd's scores were all over the place from low average on sections she refused to complete to 99.9th+ and the composite of all of that was not a DYS level score for instance, although still reasonably evidently HG. I didn't expect that to be a "wow" thing - it was just gifted and in line with where I expected all kids who were considered gifted to be at that time. Over time, I've come to realize that a lot of what they are seeing in their gifted programming are kids who have composite scores in the 70s or 80s (percentiles) on group tests with just one area in the 95th and certainly not composite scores in the top 1 or 2 percent, although achievement may be there. They also don't see a lot of 99.9th scores apparently.

I've been given the impression that our district has a high average but the distribution is not symmetric or normal about the median, such that the district's 2 sigma population is still about 130. That suggests that there's a sharp drop off in scores above the median.

In english for the not so mathy amongst us Geofizz?

Sorry. I tend to do that sometimes.

The bell curve is bent. Ours is skewed to a higher middle, but it drops off sharply at higher scores, such that the gifted range (>130) is still only ~5% of our population.

Edit: expanding on that, despite having a lot of kids who test highly, a child testing a lot higher will still be quite unusual.

Ok the private school my kids used to go to had a weird distribution - they attracted plenty of bright kids of wealthy parents, so a big bump around 120-135, but also lots of 2e kids of wealthy parents (bright kids but unable to function well maybe only one index is high and the others pull them down), the top end was also skewed a bit too, but not as much as they seemed to think :-).

This is really fascinating reading-- thank you for posting all of this, Cricket!

Our district uses a mixture of achievement tests and group ability testing to ID kids. (Why they bother, I don't really know, since anything 90th percentile on either one qualifies as "gifted" here and since that is some 30% of students, they don't really differentiate beyond that anyway, but I digress.) Our distribution here has a median of about 118, I think, which is quite high, no doubt. But it is also bimodal-- there is a regular distribution that centers at 106 or so (the state mean), and then one that centers at about 125-ish. Even so, the latter is a pretty sharp spike that encompasses (surprise, surprise) about 30% of the kids in the district. LOL. Not that it's exactly the same 30% who are ID'ed, but it's close to identical, give or take maybe 5-8 kids from that wider regular distribution.

I'm not sure what those numbers look like if they include things like WISC instead of OLSAT/CogAT.

Even so, kids like my DD are evidently quite rare-- and teachers/administrators know and acknowledge that just on the basis of her functional achievement and demeanor in person-- we have pretty much never offered outside testing but for once, and that was fairly informal and we were up front about that. Her achievement score composites are always 98-99+, but when you realize that those are 3y out of level, effectively, because of her grade accelerations...

well. We learned a long time ago that "gifted programming" isn't intended for HG+ children at all.

I, too, wonder at how such evaluations must (based on what we've seen ourselves with our DD) punish divergent thinking and reward high SES and exposure to concepts, ideas, and activities.

I also wonder whether or not ANY IQ test-- meaning either group or individual-- actually has a lock on anything other than some proxy for that quality that we'd all really like to be measuring with those tools. I seriously doubt it. But I wonder what each one DOES actually capture. It's perhaps some facet of the real (though elusive) thing, but only one facet-- which is just, as someone else noted above-- a "snapshot," and maybe it's not even a snapshot of the entire thing.

I'm not seeing the data from the study this book is citing, but there is this quote in the book WISC-IV: Clinical Assessment and Intervention (2008) that would indicate the potential for a high score on a group test but not on an individual test,

Ch. 8: The Use of the WISC-IV in Assessment and Intervention Planning for Children Who are Gifted, pg. 223 (http://books.google.com/books?hl=en...XfD1k5cUBFdXWyqL9XYQ#v=onepage&q&f=false)

Of course, as they mentioned, it could just be that the changes to the test from version III to version IV changed who was performing in the gifted range. It would have been nice if they stratified the data and looked at how the scores lined up btwn version III to version IV for kids who had taken both and not just lumped kids with gifted scores on on the WISC-III in with kids with gifted scores on group tests.

eta: there is this interesting bit of info as well - while the GDC has a self selected sample group, the group they tested on the WISC-IV when looking at its validity as a new test, got higher scores on the g loaded subtests than did the gifted group in the norming group whereas the gifted norming group did better on the speed related items than did the GDC gifted group.

http://www.gifteddevelopment.com/About_GDC/whoaregiftd.htm