Part 7: Fairness and Bias: Giving Everyone the Same Test Does Not Always Make It Fair

This is Part 7 of the 'Better Assessment Design Series:
Why Good Tests Are About More Than Questions and Scores.'
Click here for more in the series. 

Giving everyone the same assessment does not automatically make the assessment fair.

Fairness means people have a proper opportunity to demonstrate the knowledge, skill, judgement or capability being assessed.

That matters because assessments can include hidden barriers. These barriers might come from complex language, cultural assumptions, inaccessible technology, irrelevant examples, unclear scoring criteria, or background knowledge that was never meant to be assessed.

Good assessment design is not about lowering the standard.

It is about making sure the standard is measured properly.

Fairness is often misunderstood

Fairness does not mean making an assessment easy.

It does not mean reducing the required standard.

It does not mean giving everyone the same result.

Fairness means removing irrelevant barriers so the assessment measures the intended capability as cleanly as possible.

If someone fails because they do not understand the content, that is meaningful.

If someone fails because the question was written in unnecessarily complex language, that is a design problem.

Context is not the enemy

Some people worry that fair assessments need to be generic.

That is not true.

Good assessments should often be contextual.

A driving assessment should include road-based scenarios.

A science assessment should include scientific examples.

A health assessment should include relevant health situations.

A technical assessment should include authentic technical tasks.

The issue is not context.

The issue is irrelevant barriers.

The context should help measure the intended outcome, not distract from it.

Where bias enters

Bias can enter through:

• language complexity
• cultural assumptions
• examples that only suit certain contexts
• inaccessible technology
• irrelevant background knowledge
• gendered or age-based assumptions
• overly localised scenarios
• scoring criteria open to subjective interpretation

These issues can change what the assessment is really measuring.

Instead of measuring the intended capability, the assessment may start measuring familiarity, confidence, literacy, access or background experience.

Practical advice

When reviewing an assessment for fairness, ask:

• Is the language appropriate for the audience?
• Are examples culturally or contextually fair?
• Are scenarios relevant to the assessment purpose?
• Are we assuming background knowledge that was not taught or should not be required?
• Are we using idioms, slang or regional phrases?
• Is the technology accessible?
• Could people with different literacy levels understand what is being asked?
• Are accommodations required for some participants?
• Are scoring criteria clear enough to reduce assessor bias?
• Are different groups performing differently for reasons unrelated to the intended capability?

Where appropriate and lawful, results should also be reviewed by relevant groups or cohorts.

Fairness is not about lowering the bar.

It is about making sure the bar is being measured properly.

A fair assessment holds the standard while removing irrelevant barriers.

That is good measurement.

It is also good governance.

Read the next article in the series.

Rudy Crous is a corporate psychologist and the co-founder and CEO of Compono, a people and culture platform that brings hiring, engagement, learning and credentialling together in one place, adding behavioural science to the processes most HR tools only track.