Getting the Most Out of User Tests

Getting prototypes in front of users is an exciting moment in the design process. Putting the design work through its paces and seeing whether users can do what they need to is a true test of the team’s ideas and work. However, it can also be time and resource intensive to set up and run user tests.

Let’s talk about some ways to make sure that you get the most from every round of user testing you and your team runs.

Fail to Plan, Plan to Fail

The truth about research with users is that a lot of the make or break is not what happens in the room, but in the planning and set-up of the sessions.

Before you get going, make sure to create alignment around the research objectives and research questions. Think of the research objectives as the goal the research needs to achieve, and the questions as the things the test needs to answer. Getting clear on these things with the team is crucial, even if your testing is more informal or guerilla style.

The team also needs to align on the tasks that will be part of the usability test, as well as the priority order. It’s best to have a clear focus on one particular area of a site or app (such as the booking flow for an airline) rather than trying to cram too much into a test. Ensure that tasks flow from most to least important as a way to make sure the crucial pieces are covered if you run out of time in sessions.

All of the above information should be captured in a test plan and discussion guide. These documents can range from very formal to very lightweight, but it is important to have a record of the testing objectives and structure that the team aligned to. The discussion guide should provide the test facilitator with a guideline for running the sessions. I like to write guides with a structured skeleton that captures key tasks and discussion points, and then I flesh out additional prompts and questions to use as needed.

The Test is Only as Good as the Participants

Recruiting appropriate participants is the next crucial step in setting your tests up for success. Part of the research planning process should include defining who you are aiming to test with, and how many sessions you will run. The rule of thumb is between five and eight users per segment and test. This can be a good opportunity to use your personas or market demographics, if you have them, to give you some recruiting guidelines. Again, team alignment is crucial here; you want to make sure that everyone is on the same page about who to invite to participate in testing. You also want to think about representation – do you need to include users who use assistive technology? Do you have sufficient age and gender representation accounted for?

A further consideration is recruiting method – will you be paying a professional recruiter, using an online platform, or recruiting through friends and family? Each method has pros and cons, as well as various cost and resource considerations associated to it.

Recruiting with a broad set of criteria, such as people from the general population who have booked a flight online in the past year, will be much easier than specialized criteria like industry professionals, for example pilots. You should plan the length of time for the recruiting stage accordingly. As a rule of thumb, aim to have the recruiting stage complete two weeks before testing.

Rehearse, Rehearse, Rehearse

Ok, so the test plan and discussion guide is in place, the recruit is set up and in field. What now?

The time between now and the test sessions themselves is an opportunity to rehearse! The importance of this step cannot be overstated. Even for an experienced team, it is always worth having at least one full dress rehearsal to work out the kinks. My personal preference is to do one less formal internal rehearsal to try out the flow of the discussion guide and make tweaks, and a second full dress rehearsal. Ideally a colleague of family member who is not familiar with the project can play the role of test participant.

For the full dress rehearsal, it’s important is to do everything exactly as it will be on the day – including full tech and room set-up. Even if it feels unnecessary, walking through the session inevitably highlights areas that need to be finessed. It also builds confidence that you are not coming in cold on the day of the tests. I like to rehearse every last detail, including things like where the notetaker will be sitting, or where the sign in sheets will be located. When working with clients or observers, this dress rehearsal is an opportunity to prime them on what to expect. It also affords them an opportunity to share any last pieces of feedback or to voice any concerns. In order to accommodate any tweaks, the rehearsal should ideally take place about two days before the first session.

Thunderbirds are Go!

On the day of the test sessions, be sure to arrive at least an hour before the first session to set up and settle in. Arrange the final touches like making sure there is water available for participants, and the sign in sheets, non-disclosure agreements and incentives are at hand.

When the first participant walks in, it’s show time! This is where the payoff of all of the preparation and set-up of the test happens. While you can’t plan for every single eventuality when doing research, being well prepared will ensure you can meet your team’s research objectives and get the most from the sessions. During the tests, it’s all about enjoying the sessions, and rolling with it!