I'd be interested in seeing examples of good MC questions that don't have the problems I noted above; I've seen so many bad ones, the contrast would be good to see.
It would be too revealing to post mine, but let me say a bit about how I used to go about setting a multiple choice exam.
This was for a middle-years university course in a technical subject (both mathematical and human context aspects). (Not the whole of the assessment of the course, but a good chunk of it.) It was a situation in which we didn't mind the course marks having a fairly low ceiling - every year there'd be a few students who got all the MCQs right, and that was fine. Since it was the final exam, and for a class of 200 or so students, there was no expectation that it was going to give me an understanding of any individual student's progress. The MCQ answer sheets were optically read, and then I used to get not only the individual reports of what each student put and what their mark was, but also a very thorough statistical report of what happened in each question - more on that below.
- Time allowed: think of a number and double it. We used to allow an hour for 20 MCQs, of which the majority "ought" to take no more than reading time plus a few seconds of thought for a student who had mastered the material. (Dyslexic students automatically get extra time on top of that, of course - but the feedback I got was that the time we allowed was, in practice, enough for everyone, and the report validated this, see below.) I had no interest in assessing students' processing speed.
- Know what each question is trying to determine. (Sounds trivial, right? :-)
- Know what common student misconceptions are (this is the hard part!)
- Design the distractors so that a student with one of the common misconceptions has a wrong answer to plump for (it looked to me as though the AMC paper 22B posted didn't do this that well, or maybe I just didn't think of the right misconceptions in my quick look at that!)
- Make sure meta-reasoning, MCQ exam technique, doesn't easily give the answer. E.g. if you see answers 3, 30, 300, 0.3, 6, it's usually a fair bet that the answer isn't 6. Don't do that ;-)
- Going up an abstraction level, think about what the student needs to know/need to be able to do to pick out the right answer from the ones you're offering - not just about what they'd need if they had to come up with it given a blank sheet of paper. That is, watch out for whether guess-and-check will work as HK said.
- Try out the paper on a teaching assistant or colleague and if they are confused by any question, fix it!
- Use the statistical report, both checking for errors in this paper and for improving the next paper. The report told me, for example, what proportion of students chose which answer, and what proportion left the question blank. This immediately lets me check whether there was a tendency for people to run out of time - were the last questions less well or less often done than the first ones? Obviously you look carefully at the cases where a larger than usual fraction of students got the question "wrong" - was it just hard, or is there a problem with the question? Even more usefully, these reports used to summarise *which* students got a question wrong - that is, it would tell me what proportion of the students in each band (I forget, say quintile), by overall mark on the exam, got *this* question right. What you expect is that this proportion is highest for the highest quintile, lowest for the lowest. If it's pretty even across the quintiles, the question isn't doing a good job of discriminating - maybe it's too easy, or maybe it's somehow a bad question, e.g. that relies on something the course isn't supposed to be testing. If the pattern is wrong, e.g. the best students have a tendency to get it wrong more than less good students, it's likely there's another way to interpret the question that you haven't seen, or even that the "right" answer is wrong (or not the only arguable answer)! (Of course, another possibility is that the exam is testing several things, performance on which is not well correlated, but that wasn't the case for me.) This was very useful information for honing my skills in writing these questions (and it happened a couple of times that we had to disregard a question).
Anyway; we used to use this alongside some more conventional questions, but tbh, if I'd had to keep only one of those two elements, I feel the MCQ exam was the one that did a better job of reporting on students' attainment in this case. Setting the exam was a pretty skilled job, though.
