The Unintentional ‘Dieselgate’ of Usability Testing: Why Well-Tested Products Fail

Usability tests need to imitate the real world as much as possible. Because, it doesn’t matter if users excel in the lab, if they fail out in the field. Design Psychology applies its deep understanding of human psychology to create usability tests that accurately replicate real-world scenarios. The result is products that work better than your competitors.

✎ By Rune Nørager, August 2019

Key takeaways from this article:

– Usability tests needs to be carefully designed and executed by human factors specialists.

— Don’t risk product recalls or swamped support hotlines due to biased usability testing.

— Usability should be part of the business case.

Why Well-Tested Products Fail All The Time

Products fail all the time. But not because they aren’t rigorously designed and tested. On the contrary. Many products fail in the hands of users despite thorough usability testing.

But it’s not the users who are getting it wrong – it’s the usability tests that are wrongly designed. Usability tests under laboratory conditions can often help users to perform far better than they would in a real-world scenario. The result is suboptimal products and frustrated users, which ultimately threatens your company’s bottom line.

When the Lab is a Distorted Reality


A good analogy is the ‘Dieselgate’ scandal. In 2015, it was revealed that some of Volkswagen’s diesel models switched into a low emissions mode if they were being tested in a lab. This meant that the engines were highly efficient in a laboratory setting, but out on the road they emitted substantially more than the legal limit. In other words, the cars were intentionally programmed to over perform in laboratories.

Usability tests can also switch users into a high-performance mode. But unlike Dieselgate, their over performance is often unintentional. It’s an accidental impact of tests that don’t follow rigorous testing methodology, and carried out by unskilled testers.

Tests That Set Users Up to Succeed


Imagine this scenario. A usability test moderator invites a subject into a laboratory to test a product. The moderator gives the subject a detailed brief and tries to make them as comfortable as possible. The subject has all the food and drink that they need and plenty of time to use the product. The moderator asks the subject to speak their actions aloud and voice their continuous stream of thought.

The test subjects perform well and the product appears to be ready for the market. But when it ends up in the hands of real users, they start to run into usability issues – issues that were not identified during the tests.

Why it that? What has happened?

The user becomes an artificial super user

It has to do with the testing methodology - it set the users up to succeed. They weren’t hungry, thirsty or uncomfortable. They had no time pressure or lacked information. And they were protected from distractions and kept focussed on the task at hand by being made to recite their actions and thoughts out loud.

Under these conditions, the test subjects were able to draw upon their brain’s most powerful and analytical mental tools. This makes them incredibly effective problem solvers. So when they are presented with a product, they are primed to be able to decipher how best to use it – they see, think then act.

In effect - the user has become an artificial super user that is able to overcome most obstacles and is immune to a range of basic usability problems.

Testing our Thoughtless Behaviour


The above scenario is how textbooks still tell us to carry out a good usability test. But, is it actually a good way to test a product, if the product is meant to be used under pressure in the field? The short answer is ‘no’.

Many products are normally used while the user is in an intuitive state – when we presume, act, and then we think. In which case, the usability test needs to reproduce a user’s ‘normal use mode’. Otherwise, the product won’t work as well in the field as it did in the lab.

Getting test subjects to interact with a product in a normal use mode can be tricky. But it’s absolutely essential if we are to get valid results about how users will interact with the product in the real-world.

Handle the Brain with Care

At Design Psychology, we are experts at understanding how the brain works. And we apply this knowledge to create usability tests that place test subjects in a mental state that mirrors a real use scenario. We remove bias and add knowledge.

For example, to stop users drawing on their conscious analytical mental tools, we could play music, introduce random distractions, or increase their stress levels. The right strategy depends on a range of factors that are determined by the type of product, its design and its intended use.

Good Usability Testing is Business Critical


Here’s one example. Radiometer Medical came to Design Psychology with a specific goal – to demonstrate that a product works well in an emergency hospital setting. So, we designed tests that put the test subjects under stress and kept redesigning the product until it could perform under these conditions.

Radiometer Medical now have a product that is best-in-class. But they can also promote the high level of attention that was taken to ensure it is easy to use when it counts. Getting usability right means outperforming competitors – and that’s a good business case.

In other words, we are in the business of knowing what we are doing. We have different ways of designing a usability test to avoid systemic biases. And we know how to test the right type of mental tools depending on the actual intended use – not an artificial scenario with no real-world application.

Want to know more? Get in touch with Design Psychology CEO Rune Nørager.

* * *

Next Steps