Design your first Usability Test

One of the most frequently repeated rules in usability testing is:

“Testing with five users will uncover approximately 80% of usability problems.”

The idea helped establish an important principle: usability research does not always require large samples to generate valuable insights.

But the famous “five users” rule is often misunderstood.

Five participants are not a universal requirement, nor do they guarantee that a team will discover 80% of every usability problem.

The right number of participants depends on what you are trying to learn, who your users are and how varied their experiences might be.

Where Does the Five-User Rule Come From?

The principle is based on the idea that different participants will encounter some of the same usability problems.

The first participant may reveal many issues. The second will uncover some of the same problems and some new ones. As more participants are tested, the number of completely new findings often begins to decrease.

For a focused usability study involving a relatively similar group of users, a small number of participants can therefore reveal many of the most obvious problems.

This makes small studies particularly useful for iterative product development.

But this does not mean that five participants are always enough.

Sample Size Depends on the Research Question

The first question should not be:

“How many users should we test?”

It should be:

“What are we trying to learn?”

If you are testing a simple workflow with one clearly defined type of user, a small sample may reveal enough usability problems to inform the next iteration.

If your product serves very different groups, you may need participants representing each important group.

For example, a healthcare product used by doctors, nurses and administrators may require research with all three groups because their goals, knowledge and workflows are different.

Similarly, accessibility research may require participants with different access needs rather than treating them as one homogeneous user group.

The more diverse the behaviour, context and needs of your users, the less useful a single universal sample size becomes.

Multiple Small Rounds Can Be More Valuable Than One Large Study

Imagine that you have access to 15 participants.

One approach is to test the same design with all 15 people.

Another is to work iteratively:

Test with five → identify problems → improve the design → test again → learn → iterate again

The second approach often creates more value.

Why?

Participants in later rounds are testing an improved solution rather than repeatedly confirming problems the team already knows about.

This creates a continuous learning cycle:

Test → Learn → Improve → Test again

Instead of asking how many people are required to validate a finished design, teams can ask:

“Do we have enough evidence to make the next decision?”

Reliability and Repetition

When several participants independently encounter the same problem, confidence that the issue is meaningful increases.

However, frequency is not the only measure of importance.

A problem encountered by one participant could still be critical.

Imagine that only one person in a study is unable to complete a financial transaction because of an accessibility barrier.

The fact that the other participants completed the task successfully does not make the problem insignificant.

Usability findings should therefore be evaluated according to factors such as:

• Frequency
• Severity
• Impact on the user’s goal
• Business consequences
• Accessibility implications
• Likelihood of occurring in real-world use

Research is about interpreting evidence, not simply counting observations.

Recruiting the Right Participants

The quality of participants is often more important than the size of the sample.

The people taking part should represent the users whose experience you are trying to understand.

Recruitment criteria might include:

• Role or user type
• Relevant behaviours
• Experience with the product
• Familiarity with similar products
• Frequency of performing the task
• Level of specialist knowledge
• Context in which the product is used
• Relevant accessibility needs

Personas can help teams think about different user groups, but recruitment should ideally be based on real behaviours and characteristics rather than fictional profiles alone.

A beautifully designed study with the wrong participants can produce misleading conclusions.

Tasks vs Scenarios

Usability studies often ask participants to complete activities.

These can range from simple tasks to more realistic scenarios.

Tasks

A task asks the participant to perform a specific action.

For example:

Find information about the cancellation policy.

Tasks can be useful when evaluating a particular piece of functionality.

Scenarios

A scenario provides context and gives the participant a reason to perform the task.

For example:

You have booked a holiday, but your plans may change. You want to understand whether you could cancel your booking and receive a refund. Find out what options would be available to you.

The scenario creates a more realistic motivation without telling the participant exactly where to go or what to click.

Good scenarios help participants understand the situation while allowing them to decide how they would naturally approach it.

Writing Effective Usability Scenarios

A good scenario should:

• Use language familiar to the participant
• Provide enough context to understand the goal
• Describe a realistic situation
• Give the participant something meaningful to achieve
• Avoid revealing the steps required to complete the task
• Avoid using interface labels that give away the answer

For example, instead of saying:

“Click on Finance and select Car Leasing.”You might say:

“You are considering getting a new car and would prefer to spread the cost over three years. You have £1,500 available as an initial payment. Find out what options might be available to you.”

The difference is important.

The first version tests whether someone can follow instructions.

The second test is whether the interface helps them achieve a goal.

Measuring Task Performance

Teams can collect quantitative measures during usability testing, including:

• Task completion
• Time on task
• Number of errors
• Need for assistance
• Number of attempts

However, arbitrary thresholds, such as “under one minute equals two points,” should be used with caution.

A reasonable completion time depends entirely on the task.

Taking two minutes to find a phone number may indicate a serious usability problem. Taking two minutes to complete a complex financial application may represent excellent performance.

Metrics, therefore, need meaningful benchmarks.

You might compare performance against:

• The current product
• A previous design
• A competitor
• An established target
• Performance over repeated rounds

Observe Behaviour, Not Just Scores

A participant may successfully complete a task while still struggling significantly.

They might hesitate, circle around,cles or misunderstand important information before eventually finding the correct answer.

A simple “pass” would hide all of this.

During usability testing, observe:

• Where participants hesitate
• What they expect to happen
• What they misunderstand
• Where do they go first
• What they ignore
• When they express uncertainty
• When they need help

These observations often explain the numbers.

Connect Usability Testing to Continuous Discovery

Usability testing becomes particularly powerful when it is not treated as a one-off validation exercise.

In continuous discovery, teams regularly put solutions in front of customers and use what they learn to inform the next decision.

Rather than waiting until a design is almost finished and recruiting 20 participants for a large study, teams might conduct smaller studies throughout the development process.

The rhythm becomes:

Prototype → Test → Learn → Iterate

This approach changes the purpose of research.

The goal is not to prove that the design is good.

It is to discover where the team’s assumptions are wrong while those assumptions are still inexpensive to change.

So, How Many Participants Do You Need?

Sometimes five may be enough.

Sometimes, three participants may reveal the critical issue you need to address next.

Sometimes you may need 20, 50 or hundreds of participants, particularly when conducting quantitative research or comparing statistically meaningful differences.

The answer depends on the decision you need to make.

For qualitative usability testing, a useful principle is:

Start small, look for patterns, iterate, and continue testing until additional sessions no longer yield insights that meaningfully change your next decision.

The objective is not to reach a magical number of participants.

The objective is to gather enough evidence to make a better product decision, then keep learning as the product evolves.