Part 9: Data Design: If You Only Design for a Score, You Only Get a Score

This is Part 9 of the 'Better Assessment Design Series:
Why Good Tests Are About More Than Questions and Scores.'
Click here for more in the series.

If you only design for a score, you usually only get a score.

That may be enough for some assessments. But often, a total score does not tell us what we really need to know.

Someone may score 78%, but what does that mean?

Did they understand the basics?
Could they apply the knowledge?
Did they miss the most important questions?
Were they weak in one area?
Was the assessment itself working properly?

Good assessment design starts with the end report in mind.

It is not just about collecting answers. It is about designing data that can support interpretation, improvement and action.

The problem with total scores

A total score can be useful.

But a score alone is rarely enough.

Imagine someone scores 78%.

What does that actually tell us?

It may not tell us:

• which areas they struggled with
• whether they failed knowledge or application questions
• whether one cohort is underperforming
• whether one topic was poorly taught
• whether one question was confusing
• whether performance is improving over time
• whether the assessment itself is producing useful evidence

A pass or fail result tells us whether someone crossed a threshold.

It does not always tell us what they are good at, where they struggled, what support they need, or whether the learning or assessment design is working.

Better reporting leads to better decisions

Consider these two reports:

72% of participants passed.

That is useful, but limited.

Now compare it with:

Most participants understood stop sign rules, but 38% struggled to apply those rules in school zone hazard scenarios.

That second report is far more useful.

It tells us where the development need sits.

It also raises better questions:

Is the problem knowledge, understanding, application or judgement?

Is the issue with the learner, the learning content, the question or the standard?

Are certain groups struggling more than others?

Do we need to redesign the learning, rewrite the question, change the assessment structure or provide targeted support?

Good assessment data does not just describe what happened.

It helps decide what to do next.

Practical advice

Before writing the assessment, ask:

• What do we need to know after the assessment?
• What decisions will the data support?
• Who needs to use the results?
• What breakdowns will matter?
• What patterns are we trying to identify?
• What action should follow?
  

Then tag every question by:

• learning outcome, competency or standard
• learning objective
• level of thinking
• topic
• difficulty level
• question type
• risk or importance level
• context or scenario type

This allows better reporting.

For example, results can show whether people are struggling with recall, understanding, application, judgement, a specific topic, a specific outcome or high-risk content.

Data also improves the assessment

Good assessment data does not only help the person taking the test.

It helps improve the test itself.

Once results come in, assessment designers can review:

• which items were too easy
• which items were too hard
• which distractors did not work
• which questions may be confusing
• whether certain cohorts performed differently
• whether scores relate to meaningful outcomes
• whether the pass mark is appropriate

This is how assessments improve over time.

The key message is simple:

Do not wait until the assessment is finished to think about the data.

By then, it may be too late.

Good data starts before the first question is written.

Read the next article in the series.

Rudy Crous is a corporate psychologist and the co-founder and CEO of Compono, a people and culture platform that brings hiring, engagement, learning and credentialling together in one place, adding behavioural science to the processes most HR tools only track.