Diagnostic distractors
A Socratic walk-through of diagnostic distractors — reasoned out one step at a time, not lectured.
The question we started with
THE QUESTION #Why can a wrong answer option on a test tell a teacher more than the right one does?
Marking is arranged entirely around the right answer: it is what the key records, what earns the mark, what the report to parents is built from. The wrong options are scaffolding, discarded once the ticking is done. Yet a teacher who looked instead at which wrong option each child chose would often learn more from one item than the whole score sheet tells them. What is it about a wrong answer that carries information a right answer cannot?
Reasoning it through
REASONING #Ask what each response actually is, considered as a signal. A right answer on a four-option item is one outcome compatible with several very different histories: a child who understands the procedure, a child who eliminated three options while understanding nothing, a child who guessed with a one-in-four chance, and a child holding a rule attached to a cue — this phrasing means do that — with no model underneath it. The mark compresses all of those into one symbol: a threshold was crossed, and nothing about how.
Now consider a wrong answer. It is not one outcome; it is a choice among distinguishable alternatives. If each alternative is the answer produced by one specific faulty procedure, then the identity of the chosen option names the procedure.
Make that concrete. The item is 1/2 + 1/3 and the key is 5/6. Build the wrong options deliberately: 2/5 comes from adding numerators and denominators straight across; 1/3 comes from putting both over the common denominator six and forgetting to scale the numerators, giving 1/6 + 1/6; 1/6 comes from multiplying instead of adding, the easier fraction rule and the one a child who has just learned it is likeliest to over-apply. Three wrong answers, three named procedures — and a child choosing 2/5 and a child choosing 1/6 have made completely different errors that the score sheet records identically.
So is it wrongness that carries the information? Test it. Add a fourth option, 7/8, which no procedure produces. A child choosing it tells you only that they were lost. Random distractors carry nothing, so the information was never in being wrong — it was in the design of the options.
Test the other direction: is careful design sufficient? Also no. One child choosing 2/5 might have misread the operator, slipped, or clicked the wrong box. What is diagnostic is the distribution across a group: two-fifths of a class landing on 2/5 is a curriculum finding, one child landing there is a conversation. The invariant is a conjunction — a distractor is informative when it is the output of a named procedure and when it is read at the level of a group.
There is a second, empirical layer needing no theory of misconceptions at all. For each option, ask how often the strongest students chose it versus the weakest — equivalently, correlate choosing it with total score. The key should be chosen more by strong students, each distractor more by weak ones, and every violation means something specific. An option taken mostly by strong students usually means the item is mis-keyed, or the option is defensible and the key not the only right answer, or it captures an over-generalisation only students who have learned a further rule can make. An option almost nobody selects does no work: an item with four options and two dead ones behaves like a two-option item, so a blind guesser scores 50 per cent rather than 25 — the arithmetic behind a meta-analysis of the option-count studies concluding three well-written options usually do as well as four or five.
The analogy
THE ANALOGY #Think of someone driving to a house they have visited a few times. That they arrived tells you almost nothing about the map in their head; plenty of routes and plenty of guesswork end at the right door. But which wrong street they ended up on names the junction they misread — provided only one wrong reading leads to each street.
a driver takes one route on one day, whereas a student's single wrong answer may be a slip rather than a route they would take again — which is why the diagnosis lives in the pattern across a class, or across several items probing the same procedure, and never in one turn.
Clarifying the model
THE MODEL #Three refinements, one of which names a picture that is simply wrong.
The wrong picture is the word "distractor" taken literally: the idea that a wrong option's job is to distract, making the item harder by tempting the unwary. That framing licenses tricky wording, near-miss numbers and irrelevant traps — each of which lowers the scores of students who understand the content perfectly well, for reasons having nothing to do with the content. A trap-shaped option is score variation traceable to the wrong cause, the classic way an otherwise sound test stops supporting the claim made from it. A good distractor attracts one specific wrong understanding and nobody else.
The second refinement connects this to a familiar complaint: a right answer can come from a rule keyed to school cues while the child's underlying model stays untouched, which is why correct answers and correct models are separate achievements. Diagnostic distractors are the practical response — an item on which the two models give different answers can finally tell them apart, while an item both answer correctly is blind to the whole problem.
The third is an honest limit. All of this assumes the inventory of misconceptions is roughly right and that a child holding one applies it consistently. Whether naive understanding consists of coherent alternative models, or a loose kit of intuitive fragments assembled fresh depending on how the question looks, is unresolved. If the second is closer to true, option choice is evidence about a model rather than a readout of one — suggestive, worth following up, not a diagnosis on its own.
A picture of it
THE PICTURE #How to readEach point is one option of the fraction item, placed by how many weak students chose it along the bottom and how many strong students chose it up the side — the two counts a classroom item analysis produces. The key belongs top-left and is there; the two ordinary errors sit low and right, which is what a working distractor looks like; the unrelated option sits bottom-left doing no work and is the one to delete. The interesting point is the multiplication error, drifting upward because it is an over-applied rule only a student who has learned something further can make — an option in that top-right band always deserves a second look, since it may equally mean the key is wrong.
What became clearer
WHAT CLEARED #A right answer is a threshold crossing: one symbol standing in for understanding, elimination, a cue-triggered rule and luck alike. A wrong answer is a choice, and a choice among carefully built alternatives is a categorical signal naming the procedure behind it. But the information lives in the design, not the wrongness — an option no procedure generates teaches nothing — and at the level of a group rather than a child, since one wrong response is as likely a slip as a model. The options a test discards are the part that could change what gets taught next week; the score is mostly the part that cannot.
Where to go next
ONWARD #- How two-tier items, asking for an answer and then the reason, sharpen what one response can reveal.
- What ordered multiple-choice items do differently, ranking options along a path of increasing understanding.
Key terms
TERMS #| Term | What it means |
|---|---|
| Distractor | an incorrect option on a multiple-choice item; well built, it is the answer one specific wrong procedure produces. |
| Item analysis | examining how each option was chosen across ability levels, to judge the item rather than the students. |
| Point-biserial correlation | the correlation between choosing an option and total score; positive for a good key, negative for a good distractor. |
| Non-functioning distractor | an option almost nobody selects, raising the chance of a lucky guess without adding anything. |
Every term the collection defines is gathered in the glossary.