Today's Times has painful article about a nonsensical story and questions that have been used and re-used on standardized tests in at least five states. Even the guy who wrote the original story thought that the way it was used was ridiculous.

First, read the passage and look at the questions.

Then consider that this is made for EIGHTH GRADERS.

The article about the passage is here.


This explained confusion I've had for a long time about questions on multiple choice tests:




Edumacators!

Oh my! Although I've seen some doozies mentioned in my sons 4th grade study guide for the Standardized testing that make me shake my head so it is not surprising.

When ds was in 4th grade the GT teacher she was prepping the class for state testing. She was prepping them not on the material of the test but to look past all the grammar mistakes in the test. She said the GT kids get wrapped up in the test errors and leave things blank.

Pretty sad state of affairs:(

Test errors! The pretest we went through was riddled with them. I had Wolf test this year just to work on test taking, that and make sure we hit all the state standards for his actual grade. The way he does school we never know where he actually stands at grade level!

Note, though, that there is no evidence in the article that Deborah Meier has any evidence for that claim (she has "lectured and written widely about testing", not "she has managed the testing process" for example). I find it slightly implausible that these actual passages and questions are field tested before they are used - that would be a major security headache - and I find it incredible that the question-writer doesn't state the right answer along with the question, instead leaving that to be determined by the consensus in field testing. However I have an idea what may be behind the claim.

When I set multiple choice exams (which don't have to be stupid like this one!) I am sent, along with the computer mark results, a detailed report on how students performed. For each question, I am shown some statistics, including what proportion of the students in each quartile by overall mark got this question right. If that distribution is flattish, this probably isn't a good question in terms of determining what students can do (e.g., if practically everybody got it right, maybe I should ask a harder question on that topic next year). If it slopes the wrong way, with the best students overall getting the "right" answer less often than students in lower quartiles, this is a huge red flag for me to look carefully at the question and the supposed right answer, as it strongly suggests that there may be an error.

I think someone may have mistaken "check that the right answer is the consensus of the smart kids" (if it isn't, look carefully to see what happened) for "define the right answer to be the consensus of the smart kids".

That said, while I will defend MCQ testing in some circumstances, I won't defend that particular test, which seems totally weird.

Okay, I'm embarrassed, but I don't really see anything terribly wrong with these questions. And I think it was a funny story.

I think there was sufficient information in the amusing tale to answer the questions correctly. Ha ha ha, at least *I* got them all right, so perhaps that's why I think they were fine :-)

So many reading comprehension stories are incredibly boring--I would think the students would be glad to read something intended to be comical. And they should be able to analyze this type of reading, and get the joke.

PS Colinsmum , I hope you are right that Deborah Meier made that mistake, and I think your assumption it is more logical!

I think annoyed was a leap...but so would hungry be a leap. I think annoyed would be less of a leap than hungry so I would have picked that after having considered hungry. One more sentence or phrase indicting annoyance (grumbling or mumbling or something about facial expressions right before they ate him) would have cleared that right up.

The other question...I went with owl...but I could see moose for some kids...and hare because he knew he would win from the start.

I did pick those out as the hard or debated questions.

Here is an article about grading of essays on standardized tests

http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html
Facing a Robo-Grader? Just Keep Obfuscating Mellifluously
By MICHAEL WINERIP
New York Times
April 22, 2012

My comment: essay tests as a way of measuring writing skills are as problematic as multiple choice tests. I think essay exams should be used to measure content knowledge, as the essay portions of the Advanced Placement exams do.

I think a lot of the hand-wringing about the pineapple/hare story is overblown. There are only two questions of the six they had problems with... and honestly, I'd say one of them isn't a problem at all, it just takes less literal thinking that most 8th graders typically demonstrate. When the owl says, "pineapples don't have sleeves" it could be speaking figuratively, implying it can't possibly have a trick, and in this case, it didn't, so no problem.

The question about why the animals ate the pineapple is very stupid.

The criticisms of the robo-grader seem to be that length and language complexity matter, and accuracy doesn't. Since that's also a criticism of essay grading in general, I'd say the robo-grader is an excellent parody of the whole essay grading process.

http://www.nytimes.com/2005/05/04/education/04education.html

Edit: Interestingly, it seems it's the same Mr. Les Perelman making the criticisms in both articles.

I have been annoyed with lots of people, but I have never eaten them! LOL It is a weird leap in logic to suggest that the animals would eat the pineapple just because they were annoyed wih it. And since they are animals (even though they talk!) one could assume that the reason they would eat ANYTHING is just because they were hungry, but the story does not mention anything about them being hungry.


Since I am writing this post as a way of procrastinating from grading 47 college level essays, I must say that this approach does seem to exist beyond the standardized testing world and this article gave me a little bit of insight into why my students might think that a long essay with big words is "good" writing. I don't think I saw mention in either article of the elements of a well-structured essay or a solid argument. Even if your facts are off, you can make a strong, well formed argument if you've been taught that skill. For example, my students are supposed to make policy recommendations to address a social problem of their choice. I know I will see MAYBE a quarter of the essays that present an argument for their recommendations as instructed rather than a gathering of opinionated statements on the subject.

Best classes I took in high school were two years of journalism that taught me to be clear and concise, gave me the skills to write a well structured essay on ANY topic, and that a well communicated idea is not necessarily the one that uses the biggest words and takes the longest to write.

I think that two major problems in US education are that 1. we've tried to industrialize education and assessment, and 2. and very few people question this idea.

What were those questions in pineapple story actually measuring? What's meaningful about the outcome if the animals had cheered for the hare instead? Isn't it bloody obvious? And what's so wise about an owl saying that pineapples don't have sleeves? Isn't that bloody obvious, too? Besides, the moose was speaking metaphorically and the owl didn't get that. Pineapples don't have mouths or nervous systems either, yet this one could talk. It seems reasonable that if he could talk with no mouth, he could have found a way to win a race with no, emm, sleeves. Wouldn't it have been better to ask students to write a composition on absurdity and this piece?

IMO, the scan-and-score tests favored in the United States don't measure a student's ability to think about something and draw conclusions from ideas. They just measure how quickly he can yank answers out of a passage. Do we want to educate our kids so that they can consider many facets of an idea, or just teach them how to move quickly from factoid to factoid?

Compare with the Irish Junior Certificate. Students take this test when they're 14 or 15 (the pineapple story is aimed at US 14 year olds). This is only the ordinary level paper (but check sections 4 & 5). Here's an advanced-level exam aimed at students we might call honor students.

There's a world of difference between the US tests and the Junior Cert exams. People have to read the Junior Cert papers and think about them a bit (Ireland doesn't use robograders). But we don't seem to be willing to do this in America. We seem to carry our national pastime of being in a hurry down to the level of young kids. IMO, we're cheating kids and our society by not teaching people how to slow down and think about something.

But a citizenry with critical thinking and reading skills would be a huge drag on the media, politicians and corporations. They might have to make their reports more accurate and their arguments more logical! Much easier to deal with a population that is satisfied with colorful factoids.