Multiple-choice questions are commonly used for learning assessments because they are easy to grade and are more objectively graded than some other question formats. But I’ve heard people complain that multiple-choice questions shouldn’t be used because they only measure recall of content.
Whaaaat? That’s like saying PowerPoint is bad because slides too often have small type that isn’t easily read. Small and unreadable type on slides isn’t the fault of PowerPoint but of the person designing the slides!
Similarly, multiple-choice questions that only measure recall of content is the fault of the multiple-choice question writer. Writing good multiple-choice questions is a critical skill for people who design instruction!
Research shows that multiple-choice questions can, when well-written, simulate performance, especially if we are asking about decisions and steps to take. Well-written multiple-choice questions have a lot of advantages as an assessment method, including the fact that we can assess more content in a shorter amount of time than we can using other assessment formats, such as essay questions.
But most aren’t well-written. Research shows that too many multiple-choice questions have a host of flaws. And those flawed multiple-choice questions damage assessments, confuse participants, and cause a lot of other problems for people and organizations, including the potential for legal problems.
In this article, I’ll discuss three common multiple-choice question flaws, show an example of each, and explain what we should do instead.
1: Answer choices are not all the same length
More experienced test-takers do what they can to eliminate some of the answer choices, especially when they are unsure. This gives them a better chance of guessing the right answer. And—they know that the longest answer is often the correct one. That’s because we often include more detail in the correct answer. More details usually mean a longer answer choice.
Here’s an example of a question where the longest answer has more detail and is correct.
You are using your outdoor word-burning ceramic fire pit. Which of the following should you do to safely operate the fire pit? (Select the best answer.)
- Burn pine or other soft woods in the fire pit.
- Make sure that there aren’t any combustible materials, such as dead vegetation, adjacent to the fire pit.
- Douse the fire with water from the hose to put it out.
The correct answer is b. It is also the answer that is longest and has the most detail.
FIX: Write all answers choices so they are about the same length. Sometimes that’s really, really hard to do. In that case, consider to writing two answers that are shorter and two answers that are longer. But if the longer answers are more detailed, people know they’re more likely correct. In that case, they’ll be able to eliminate two of the four answers, making it easier to guess.
2: Including unnecessary details
Questions should be focused, concise, and clear. If they include unnecessary information, people will try their hardest to make sense of how that information relates to the answer choices. This increases the mental effort of the question in unnecessary ways and diminishes the question’s value.
You want to use your wood-burning fire pit for warmth on chilly evenings and to roast marshmallows. To take advantage of the shade under the gazebo, could you safely place the fire pit underneath? (Select the best answer.)
- Yes, if the gazebo ceiling is high enough to allow for heat dispersal
- Yes, if you use screening material over top of the fire pit to reduce sparks
- No, because the smoke and fumes from the fire pit may damage the gazebo ceiling
- No, because only propane fire pits may be used in a gazebo
The correct answer is a, and it is the only answer that correctly answers how to safely operate a fire pit under a gazebo. The unnecessary detail about warmth and roasting marshmallow muddies the question.
FIX: Keep all answer choices similar in length and level of detail.
3: Using “All of the above” and “None of the above” as answer choices
Which of the following are good woods to burn in a fire pit? (Select the correct answer.)
- Oak
- Hickory
- Apple
- All of the above
The correct answer is d. All or none of the above may seem like an easy way to add another answer, but they are poor choices. If someone can identify a single answer choice as correct, they know that “none of the above” isn’t correct. If they can identify a single choice as incorrect, they know that “all of the above” isn’t correct. When people can eliminate answers, they can more easily get the question correct even if they don’t know the answer.
FIX: Don’t use “All of the above” or “None of the above” as answer choices.
Write better questions
Because poorly written multiple-choice questions are a widespread problem, I decided to do something about it. My first Deeper Learning at Work online course is on writing multiple-choice questions.
References
Abedi, J. (2006). “Language issues in item development.” In S. M. Downing and T. M. Haladyna (Eds.), Handbook of Test Development. Routledge.
Downing, S.M. (2005). “The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education.” Advances in Health Sciences Education: Theory and Practice, 10(2), 133–143.
Haladyna, T.M. & Downing, S.M. (1989). “A taxonomy of multiple-choice item writing rules.” Applied Measurement in Education, 2(1), 37–50.
Haladyna, T.M. & Downing, S.M. (1989). “Validity of a taxonomy of multiple-choice item writing rules.” Applied Measurement in Education, 2(1), 51–78.
Marsh, E. J. & Cantor, A. D. (2014). “Learning from the test: Dos and don’ts for using multiple-choice tests.” In McDaniel, M. A., Frey, R. F., Fitzpatrick, S. M., & Roediger, H. L. (Eds.), Integrating Cognitive Science with Innovative Teaching in STEM Disciplines. Washington University, Saint Louis, Missouri.
Nedeau-Cayo, R., Laughlin, D., Rus, L. Hall, J. (2013). “Assessment of item-writing flaws in multiple-choice questions.” Journal for Nurses in Professional Development, 29(2), 52-57.
Shrock, S. A. & Coscarelli, W. C. C. (1989). Criterion-referenced test development. Reading, MA: Addison-Wesley.
Tarrant, M., Knierim, A., Hayesm S.K., & Ware, J. (2006). “The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments.” Nurse Education Today, 26(8), 662–671.