Part 6: Item Design: Every Question Needs to Earn Its Place

This is Part 6 of the 'Better Assessment Design Series:
Why Good Tests Are About More Than Questions and Scores.'
Click here for more in the series.

A test is only as strong as the questions inside it.

Many assessments look complete because they include a set of questions, answer options and scoring rules. But if the questions are weak, unclear, trivial, poorly mapped or badly written, the assessment result will be weak as well.

This is why every question needs a job.

A good assessment item should not be included simply because it is related to the topic. It should provide evidence against a learning outcome, competency, standard or decision.

Good assessment design is not just about having enough questions.

It is about making sure every question earns its place.

The danger of “related” questions

A question can be related to the topic and still not be useful.

For example:

What colour is a stop sign?

This is related to road signs. It is not completely irrelevant. But does it provide strong evidence of safe road behaviour?

Probably not.

A stronger question would be:

You stop at a stop sign, but your view is blocked by a parked car. What should you do before entering the intersection?

This question gives better evidence because it asks the person to apply safe behaviour in context.

The issue is not whether a question is connected to the topic.

The issue is whether it provides useful evidence against the intended outcome.

Why experts sometimes write weak questions

Subject matter experts are essential to assessment design.

They understand the content, the risks, the standards and the real-world situations.

But being an expert in a subject does not automatically make someone an expert in assessment writing.

Experts may write questions that are:

• too technical
• too detailed
• too focused on rare exceptions
• too dependent on insider knowledge
• too much like textbook or policy wording
• too focused on what is interesting rather than what is important

That is why good item design needs both expertise and discipline.

The expert helps define what matters.

The assessment designer helps turn that into clear, valid evidence.

Common item design problems

Poor questions often include:

• more than one correct answer
• no clearly correct answer
• answer options that overlap
• obviously wrong distractors
• trick wording
• double negatives
• irrelevant trivia
• two ideas in one question
• questions that are too broad
• questions not mapped to a learning outcome
• questions written at the wrong level of thinking

One weak question may not ruin an assessment.

But a pattern of weak questions creates weak evidence.

And weak evidence leads to weak decisions.

Practical advice

For every question, ask:

• What outcome, competency or standard does this item assess?
• What is the correct answer?
• Why is it correct?
• Why are the other options wrong?
• Is there only one best answer?
• Is the question testing something important?
• Is it pitched at the right level of thinking?
• Is it clear and readable?
• Does it reflect real-world use where appropriate?
• What evidence does this question provide?

A simple rule is:

One question, one purpose, one best answer.

For multiple-choice questions:

• keep the stem clear
• make distractors plausible
• avoid obviously silly options
• avoid “all of the above” where possible
• avoid “none of the above” where possible
• make answer options similar in length and style
• do not give away the answer through wording

Good item design is not about being clever.

It is about collecting clean evidence.

Every question should earn its place.

Read the next article in the series.

Rudy Crous is a corporate psychologist and the co-founder and CEO of Compono, a people and culture platform that brings hiring, engagement, learning and credentialling together in one place, adding behavioural science to the processes most HR tools only track.