Gifted Issues Discussion homepage Forums Identification, Testing & Assessment Pineapples don't have sleeves!

Today's Times has painful article about a nonsensical story and questions that have been used and re-used on standardized tests in at least five states. Even the guy who wrote the original story thought that the way it was used was ridiculous.

First, read the passage and look at the questions.

Then consider that this is made for EIGHTH GRADERS.

The article about the passage is here.


This explained confusion I've had for a long time about questions on multiple choice tests:




Edumacators!

Oh my! Although I've seen some doozies mentioned in my sons 4th grade study guide for the Standardized testing that make me shake my head so it is not surprising.

When ds was in 4th grade the GT teacher she was prepping the class for state testing. She was prepping them not on the material of the test but to look past all the grammar mistakes in the test. She said the GT kids get wrapped up in the test errors and leave things blank.

Pretty sad state of affairs:(

Test errors! The pretest we went through was riddled with them. I had Wolf test this year just to work on test taking, that and make sure we hit all the state standards for his actual grade. The way he does school we never know where he actually stands at grade level!

Note, though, that there is no evidence in the article that Deborah Meier has any evidence for that claim (she has "lectured and written widely about testing", not "she has managed the testing process" for example). I find it slightly implausible that these actual passages and questions are field tested before they are used - that would be a major security headache - and I find it incredible that the question-writer doesn't state the right answer along with the question, instead leaving that to be determined by the consensus in field testing. However I have an idea what may be behind the claim.

When I set multiple choice exams (which don't have to be stupid like this one!) I am sent, along with the computer mark results, a detailed report on how students performed. For each question, I am shown some statistics, including what proportion of the students in each quartile by overall mark got this question right. If that distribution is flattish, this probably isn't a good question in terms of determining what students can do (e.g., if practically everybody got it right, maybe I should ask a harder question on that topic next year). If it slopes the wrong way, with the best students overall getting the "right" answer less often than students in lower quartiles, this is a huge red flag for me to look carefully at the question and the supposed right answer, as it strongly suggests that there may be an error.

I think someone may have mistaken "check that the right answer is the consensus of the smart kids" (if it isn't, look carefully to see what happened) for "define the right answer to be the consensus of the smart kids".

That said, while I will defend MCQ testing in some circumstances, I won't defend that particular test, which seems totally weird.

Okay, I'm embarrassed, but I don't really see anything terribly wrong with these questions. And I think it was a funny story.

I think there was sufficient information in the amusing tale to answer the questions correctly. Ha ha ha, at least *I* got them all right, so perhaps that's why I think they were fine :-)

So many reading comprehension stories are incredibly boring--I would think the students would be glad to read something intended to be comical. And they should be able to analyze this type of reading, and get the joke.

PS Colinsmum , I hope you are right that Deborah Meier made that mistake, and I think your assumption it is more logical!

I think annoyed was a leap...but so would hungry be a leap. I think annoyed would be less of a leap than hungry so I would have picked that after having considered hungry. One more sentence or phrase indicting annoyance (grumbling or mumbling or something about facial expressions right before they ate him) would have cleared that right up.

The other question...I went with owl...but I could see moose for some kids...and hare because he knew he would win from the start.

I did pick those out as the hard or debated questions.

Here is an article about grading of essays on standardized tests

http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html
Facing a Robo-Grader? Just Keep Obfuscating Mellifluously
By MICHAEL WINERIP
New York Times
April 22, 2012

My comment: essay tests as a way of measuring writing skills are as problematic as multiple choice tests. I think essay exams should be used to measure content knowledge, as the essay portions of the Advanced Placement exams do.

I think a lot of the hand-wringing about the pineapple/hare story is overblown. There are only two questions of the six they had problems with... and honestly, I'd say one of them isn't a problem at all, it just takes less literal thinking that most 8th graders typically demonstrate. When the owl says, "pineapples don't have sleeves" it could be speaking figuratively, implying it can't possibly have a trick, and in this case, it didn't, so no problem.

The question about why the animals ate the pineapple is very stupid.

The criticisms of the robo-grader seem to be that length and language complexity matter, and accuracy doesn't. Since that's also a criticism of essay grading in general, I'd say the robo-grader is an excellent parody of the whole essay grading process.

http://www.nytimes.com/2005/05/04/education/04education.html

Edit: Interestingly, it seems it's the same Mr. Les Perelman making the criticisms in both articles.