How To Create Better Diary Studies, With Workflow and Task Analysis.

Diary studies are often used, expecting that the rich qualitative data will reveal new insights. But they can also be highly time-consuming and effortful for participants to fill in and for researchers to analyse. In this article we share how we used a structured diary method to find out what really matters to consumers about vacuum cleaners.

✎ By Demet Soyyilmaz and David Hvidbak Østergaard, April 2022

Key takeaways from this article:

– With a structured, workflow approach, diary methods become easier for participants and for those analysing it

— To provide users with good consumer guidance, we need to know what problems actually matter in daily use

— The UX Value Equation, can help us distinguish between utility and usability aspects of a product

What Is a Good Vacuum Cleaner, and How Do We Test It?

Design Psychology worked with a consumer organisation to help them gain insights from vacuum cleaner users across Europe. Our objective was to help them determine what matters most to consumers about vacuum cleaners in actual, daily use and how they can centre their testing around these most important factors.

The Setup of the Diary Study:

To determine what matters most when using a vacuum cleaner, we took the test to homes across Europe. We asked people to keep a diary about their experience of using their vacuum cleaner.  In addition, we asked for video and photos to fully understand what constituted the best and worst experiences. 

You might think, that’s a lot of data! And yes, almost any diary study will lead to large amounts of data to analyse. But with proper preparation (i.e. tasks analysis), we created a Workflow Diary Tool that both supported the participants in what to write about and helped us navigate the vast dataset.

What is the Workflow Diary Tool?

To structure the study, we used task analysis to build a workflow diary tool.

That is, for each of the four types of vacuum cleaners that was a part of the study, we broke down the workflow in to concrete, small interaction steps that users engage in when using vacuum cleaners.

This included some high-level tasks such as storage, setup and use, and maintenance. Each of these high-level tasks consists of smaller subtasks, such as ‘pickup vacuum cleaner’, ‘transport it to the cleaning area’, ‘take out the power cord’, ‘plug it in’ and more.

The diary packet we send to our participants (see examples on the right), was structured so that the questions and layout would capture the user experience at all task and subtask levels. We also asked the participants to film and share some video clips of their daily use, storage and maintenance of their vacuum cleaners.

The workflow diary tool aimed to probe users to share as much as possible with a guided format. This structure prevented users from being overly descriptive and repetitive, which are common challenges in conventional diary studies. The workflow diary tool enabled users to report quantitative and qualitative data with clear instructions and user-friendly administration.

The Workflow Diary Tool can be applied to any product category.

Defining the User Experience of Vacuum Cleaners

We used the ‘UX Value equation’ as our analytical approach to the diary study.

In this approach, user Experience consists of two major facets, Usability and Utility. When applied to vacuum cleaners, Utility is a summation of both the core technology (vacuuming engine), user features (e.g. cable wind-up) and core features (e.g. variable suction power).

Usability, on the other hand, delineates details such as ergonomics of handles, user interaction with indicators (e.g. to clean filters or change bags), the flexibility of the vacuum’s hard parts, etc.

All of these aspects were captured, analysed, and transformed into measurable metrics for the study.

The UX Value Equation helped us prioritise the most important aspects of vacuum cleaners to consumers.

Insights From the Workflow Diary Method

The workflow diary method can provide many different types of insights, for example:

1. Finding mismatches between which factors consumers believe are most important while purchasing, and which factors are actually most important to the user experience after purchasing.

2. Defining aspects of use, that are special pain points for different types of products, and can only be discovered through long-term use (e.g. cleaning robot vacuum round brushes).

3. Highlighting areas in long-term use, where consumers lack knowledge or misunderstand the product.

4. Discover ‘mental models’ about the product, such as how consumers think about, talk about and perceive the product or product category. Mental models can help improve user tests and product development.

Empowering Consumers With User Insights

Applying the workflow diary tool to vacuum cleaners revealed substantial insights into what vacuum cleaner consumers value at daily use that they were not qualified to consider before purchasing a new product.

It is crucial to discover what really matters to users to develop optimal products, and help them make more informed choices at purchase.

* * *

Next Steps