User Testing Moves Online

This article describes our experience with online user testing, why it is a good idea and how to do it.

User testing for websites moves online

User testing is one of the most important practices in website design and usability. It remains by far our best method for finding out how users respond in depth and detail to our work.

Every user testing process we have run has generated important new insights and these, in turn, have led to substantial, often radical, improvements in our work.

In 2020 COVID-19 arrived in the UK and the country went into lockdown. By unfortunate coincidence we were booked to run a series of user tests at a lab in London just after the start of lockdown. Of course, the tests had to be cancelled.

For a time we thought we’d have to wait until normal life resumed before we could do any more user testing. But then, as people became accustomed to using Zoom for all kinds of meetings, we realised that we could resume user testing in a new pattern, online.

Better, less expensive, more flexible

Pursuing this idea, we learned that user testing for websites, which all used to be done face-to-face in a user testing lab, can now be done online using Zoom (or similar online video conferencing system). And more than that, while the results are at least as good, the process can be better, less expensive and more convenient:

• Better because using Zoom you can easily schedule and run each session and then get a complete digital recording.
• Less expensive because you no longer need to hire a user testing lab or pay test subjects to travel to your session.
• More flexible because you can schedule the sessions to suit each test subject’s convenience rather than the schedule of the user testing lab facilities. One consequence of this is that we have been able to run user tests with subjects in different countries without anyone having to do any travelling.

During 2021 we did ten online user testing sessions for our work on the Visual Commentary on Scripture (www.thevcs.org). Half the subjects were based in the UK and half in the US. The sessions were highly productive and the insights we gathered have helped us to design and implement radical changes to the organisation of the site’s content and its presentation. Without those sessions we would never have understood the many subtle (and some not-so subtle) ways in which our target audience wanted our work to improve.

If that sounds useful, then read on to find out how it’s done. The process described is what we are now using. It is exactly what we did for the VCS test sessions described above.

Note: We are not saying that we will never do ‘in person’ user testing again. There are still occasions where the immediacy of group sessions with discussion in real-time will be desirable. But for the majority of applications, the online alternative described here is likely to be preferable.

How to run an online user testing process

Note: what follows describes how to user test a website, but the process can be easily adapted to work for other kinds of products and services.

Plan the process including realistic timescales. It can take quite a while, especially if the test sessions have to be spread out.

Recruit testers — it is important that they offer a diversity of viewpoints and, taken as a group, they should represent a broad cross-section of the target audience(s).

Aim for no less than five test sessions (no more than 8). Aim to test with people individually or in pairs if that seems appropriate — this method will work either way.

Write a test script based on the project’s strategic mission and the tasks that you want people to be able to accomplish. A recent script was divided into sections as follows:

• Pre-amble — technical set-up and protocol for the session including permission to record
• Introduction — about the subject and what they do
• Current behaviour — what they need in relation to the site being tested and how they currently meet their needs online
• Exploration of the website — a variety of tasks to walk through
• Wrap-up — final questions and overall impressions

The test session should be planned and scheduled to last for about an hour. More is too long, less is not enough time.

Prepare collateral for presentation. If the website itself is not ready to test then you can use design mock-ups. Don’t use wireframes — they are generally too abstract for most people to understand.

Questions should elicit:

• Background information about the test subject;
• General reactions to the collateral;
• Impressions of the design and quality of the work;
• Comments about the connection between their own situation, the project’s mission and the collateral presented.
• Does the mission resonate with their situation?
• If so, does the collateral show that the project is on the right track?
• How could it be improved?
• Do they understand how they could accomplish the relevant tasks?
• etc…

Schedule the sessions using Zoom.

Run the sessions — walking each of test subjects through the script in the same way.

Record the sessions.

Use Trint (www.trint.com) or similar online transcription service to get draft transcripts.

Using the draft transcripts, prepare individual notes for each test subject.

Based on the individual notes, prepare a summary of findings with recommendations.

Fold those recommendations into the project backlog.

Implement changes accordingly.

Rinse and repeat on a regular basis.

Summary

User testing remains one of our best techniques for finding out how people respond to a website. User testing that was done face-to-face can now be done online. The new online process is in many ways better than the original.

If you want to know more about any of the subjects touched on in this article, please get in touch via our website at www.cogapp.com.