user testing

What to Expect in the Usability Report

By | implementation, user testing | No Comments

From the first four posts in the series you now know the What, When, How, and Why of Usability Testing, centered specifically on User Testing.

When the test is done you will need to analyze the data from the test and write up your learnings. This is the usability report. If you’re working with a researcher like myself, then this is the deliverable that you will receive which breaks down the test and ultimately makes the recommendations for next steps. If you’re conducting the test yourself you should still take the time to write down your findings, even if it’s not quite as formal.

The summary should be a one page overview of the test that the stakeholder can review and have all the relevant information. This includes why you chose to perform the test, how many participants were tested and what persona(s) they fall into, the date(s) the test was performed on, and the key takeaways.

The introduction is a more in depth continuation of the summary explaining an overview of the decisions that brought you to the particulars of the usability test with details. An overview of usability testing and the steps you took prior to conducting the test.

Key Hypotheses
List the 1-3 hypotheses you made prior to executing the test. For example:

  • Users will be able to easily navigate from the welcome email into their account
  • Users will be able to buy a shirt after entering the home page
  • Users will be able to find and contact support

Now that you’ve conducted the test, what goals are you hoping to achieve?

  • Through a series of iterations, we will improve the comprehension of the data being presented by 80%.
  • Through a series of iterations, we will lower the user’s lostness score to below .2.

To read more about lostness score, read this Medium post by Shane Doyle.

Give an overview of your participants and include their demographic data and interview questions. If you have enough disparity add some pretty charts and graphs to show how they compare to one another.

Methodology & Scenarios
Some people seperate these into two sections, but I prefer to stick them together. Write up which testing methodology you used and then in a number list write up the scenarios you asked the users to walk through. You can copy them directly from the script you wrote for the test.

This section is broken down into four major parts:

  • Task Completion – A write up of each task and what the rate of success was for each user. Normally in the form of a table, followed up by detailed paragraphs.
  • Errors – any bugs that were found during the test.
  • Key Takeaways – Same as in your summary, the key learnings that you found in the test.
  • Notable quotes – one of my favorite parts. All those gems that your users spit out without even realizing how valuable they are. List them here. Mine tend to be a little longer than others I’ve seen because I’d rather have too many to pick from in the future than not enough.

The longest section by far. This is where you list all the problems you came across (beyond the key takeaways into the nitty gritty) and your recommendations about what to do about them. How I typically list them is:

  • State the problem – one sentence explaining the problem.  I also include a color-coded severity of the problem Low (green), Medium (orange) and High(red)
  • A screenshot for reference
  • Detail – an in-depth overview of the problem with reference to the participants issues
  • Recommendation – may include a simple wireframe to go along with the text recommendation
  • Additional comments (optional) – if there’s something beyond the recommendation that I feel needs to be addressed I’ll put it here

You repeat this until you’ve covered everything that has come up in the test.

This is usually 2-3 paragraphs analyzing the effectiveness of the test and suggestions for any next steps after the recommendations are implemented.

There you have it, a report to go along with all your hard work. As I recommended in the How portion of this series, UXPin has some great templates, including one for the report.

Next up: Pizza.

Why To Adopt Usability Testing

By | user testing | No Comments

I get jolt of pleasure followed by a knowing smile from the reaction stakeholders have after their first customer interview experience or user testing session. It’s an a-ha moment. Happiness and then a flurry of the learnings and improvements they just picked up. And that’s after just the first one.

You want to run usability tests to optimize your user experience in order to optimize your bottom line. Creating a site or app that is intuitive for your user in turn creates a happier user. You need to understand the path that users are taking in order to achieve success and smooth out the edges. The less they need to think, the happier they’ll be and the more they’ll buy. Remember, product success is when your customer is successful.

My favorite learnings are the ones which leave me feeling a little stupid. A user will be walking through the site which I know very well at this point and will ask something like, “I understand that this button adds it to my cart, but why can’t I immediately go to my cart?” In building a product you can start to have tunnel vision towards your own ideas, but a good user test will help you to put a wider lens on your camera and expand the vision of what you’re building.

The biggest example is the story of the $300 Million button. Jared Spool explains how removing a user’s need to create an account in order to purchase from a large online retailer increased yearly sales by $300 million. He found out that the original hypothesis that users wouldn’t mind registering for an account was wrong and most did so with a sense of despair. They removed the register button, replaced it with a continue button and number of customers purchasing went up by 45%.

This is the power of usability testing.

One more in this series, centered around the usability testing report. Then onto a case study about ordering pizza online from three of the biggest chains.

Want to read more about usability testing? Leave a comment and let me know.

How to Run Your First Usability Test

By | implementation, user testing | No Comments

In my last two posts I wrote about the What and When of usability testing. Today I’ll be running through the How for first time testers.

There are a lot of different ways to study the usability of your product and depending on the situation you want to adjust your approach accordingly. The most useful place to start is with user testing and the great part is that there are some quick and easy steps so that  anyone can do it.

From a high level, you’re going to get users that are representative of real-life users, give them tasks to complete that real-life users would complete and watch what they do and how they do it.  Take notes, communicate what you see to your team and already you’ll have a better product. To do a slightly more formal study dive into the steps below.

Define Your Goal
First, what’s your goal? Just as you’ve defined the goal of your product, in turn you need to understand your goal of the test you’re going to perform before anything else. For example:

Can users find out about my consulting services?

Which method do users use to find a product in order to put it in the cart?

The important part in defining this is to create focus around the most important aspect of your product to test. The wider your testing lens (the more things you try to test) the more time a user has to begin learning your UI and creating skewed results. Remember the goal is to get a user’s initial reaction, not hoping they figure your UI out.

Create the Tasks
The next step is to define the tasks that you want the users to perform. What steps do you need them to take in order to achieve your goal?

Goal: Can users find out about my consulting services?
Task: Use the website to find out about any consulting services.  

Goal: Which method do users use to find a product in order to put it in the cart?
Task: Use the website to purchase a shirt for under $30.

Define Your Users
You need to understand who is going to be using your product. List out their primary characteristics so that when you go out to recruit subjects to help with your testing you can try to match as closely as possible to what a real-life user looks like. Some products have multiple users who will use the product differently. If that’s the case make sure to focus on the correct personas for the test you’re running. If multiple groups will need to achieve the goal you’ve laid out, make sure to get representatives from each set.

Write Your Plan
There are many good reasons to write your plan down, but the two most important are to make sure the stakeholders understand what you’re planning on doing and are on board and to have a clear layout throughout the testing.  

I still use this sample plan as a template when I conduct a user test:

And for the script part I always start with this:

Although over time mine has evolved into more of my voice and shifts based on what type of product I’m testing.

In addition UXPin has put together some great documents which you can find here, including a video consent form and a checklist for test day. They are a good starting point but definitely recommend making them your own.

Recruit Users
This is often viewed as one of the most difficult parts of the overall process. Finding the right kind of users (as you’ve defined above), scheduling times and preparing the documentation. For this, try to find users that are as close to real-life users as possible but don’t be super strict. Find friends, family or co-workers that fit the bill and ask them to do you a favor, especially if this is your first time through. Also, if you have existing users that’s how I’ve done the majority of my testing throughout my career.  They’re often more than happy to contribute to helping especially if they’re a fan.  

This article does a good job of laying out ways to recruit different types of users.

Conduct The Test
You can run the test face-to-face or over a screen sharing service. I’ve used in the past and in my most recent study used Go To Meeting, which worked well and was cheap.

Regardless of your proximity, address the user by name and thank them for their time. Make sure they are comfortable with the platform and then run through your script.

When it’s the user’s time to shine, say as little as possible and don’t answer any questions that might guide them to an solution that you’re hoping to find through the testing.  Let them know you’ll answer that question when the test is done or that you want to see how they work it out for themselves.

Once you’ve gone through all of the scenarios, ask any questions through what they saw or as follow up to the tasks they’ve just performed and make sure to give the user time to ask any questions of their own.

Write the Report
Using your notes and reviewing the video make an outline of the improvements that could be made to the site. Mark each change with a priority level and include both written and sketched suggestions so that you can add it to your next sprint or planning cycle. If it’s helpful you can even edit sections of the video to share with the engineering or designers so they can see the problem first hand.

I like to create a folder on Google Drive and include all the videos, the initial write-up and the final report along with a file linking to any stories or tickets I’ve created.  

As I said early, there are many other types of usability tests which I will go over in a future post, but for now you’re armed to go and run a user test and improve your product.

What else would you like to know about user testing or usability studies? Let me know in the comments.

Are you interested in having me run a usability study for you? Contact me.

When Should I Do Usability Testing?

By | product, strategy, user testing | No Comments

In the last post I explained what a usability study is. If you haven’t read that, check it out here.  

Today I wanted to focus on when to do a usability study, but before we dive into that let me first address what types of products benefit from a usability study.

When I talk about usability studies I’m typically focused on tech products which I group into websites, web applications, native applications or mobile apps. I do believe there is benefit in bringing in the user for other types of products during the entire lifecycle and getting feedback along the way as you design and build, whether that be something physical, like the next great widget or something like a book, which Ash Maurya lays out in great detail in his book Running Lean (for that very book).

Bringing it back to tech products my answer would be that they all benefit from usability studies. I like to focus on the low and high end of complexity as use cases and letting the middle sort itself out, so in that spirit let’s break it down:

Low complexity
A portfolio website to show off your artwork and increase consulting work. Surely you don’t need to do any usability testing. It’s straightforward, here’s some beautiful design work and here’s how to get in touch with you. Boom, done. My counter argument to that line of thinking is, what if you could increase the amount of consulting work you get by 10% with a few hours of work? How about 50%? By putting the site in front of people with the task of hiring a design consultant you could discover that they want to fill out a form rather than call or email you. That small change could increase your rate of return.

High complexity
An advertising platform that allows advertisers and publishers to manage their accounts in real time. This would be a web application with a mobile component that allows  both external and internal users different levels of access. With this type of product you would have many levels of usability testing. From a persona level you have advertisers and publishers, all of which are probably not the same and would need to be broken down into sub groups. Even within those personas it’s often the case that users will need different levels of access, such as the Sales Executive and the Ad Buyer. That’s without yet considering how the internal account managers will need to impersonate users and have their own switches and levers.  

Putting these products in front of users is crucial to saving time and money as you’re building and maximizing profits after it hits the market.  

But when do you put it in front of users? When is the right time to do usability testing?

The short answer is, throughout your entire process, from discovery to design, through development and even after launch.

In Tom Chi’s talk on Rapid Iteration (which I posted back in December) he says, “For everyone who builds products, the real medium is human behaviour in the real world, and your purpose should be…to see that human behaviour modified by what you build.”

The longer answer is that the more usability testing you can achieve during your product creation, the better. In Tom Chi’s talk he mentions a feedback loop of only a few days, constantly giving feedback to the design and engineering teams. That is a high bar to reach for indeed.

Instead, breaking the project down into Discovery, Design, Development and Launch we can start to identify the points at which usability tests can be effective, even if we aren’t on a bi-weekly cycle with our feedback loop.

Product discovery starts with understanding what you’re hoping to achieve. See my post on setting your product vision. Involving users at this stage typically means conducting user interviews or nothing at all. One thing you could do as you’re qualifying your intuition is put users in front of similar products and gain insight from how they interact.

During the design phase I encourage iterative design and adding in parallel design if you have the time. Iterative design is the process of taking user feedback on a design and updating it with marked usability improvement with each iteration that you do.  Parallel design starts with multiple concepts and takes the best from all three to end up at 1 design to start the iterative process from.

You want to work through as many cycles as you can, continually gaining insight from users, before a line of code is ever written. In this great article on the Nielsen Norman Group website it goes through the details of each of these projects and state “there’s no one perfect user interface design, and you can’t get good usability by simply shipping your one best idea. You have to try (and test) multiple design ideas.”

During this phase UX, design and engineering should be talking every day about progress of the project and UX should be thinking about what they’re going to test on a weekly basis. In the video from Tom Chi above his team does it twice a week. Do what you can, if you can only do it once every two weeks it’s better than not at all. By taking what engineering is building and putting it in front of users and finding the flaws (and really awesome stuff) as you’re building it will save you from huge rewrites and updates after launch which saves your team huge amounts of time and money.

As I wrote about a few days ago, when you launch you’re not finished, keep pushing towards your goal that you set up at the beginning of the project and while now you will have quantitative data coming in through the many users you have hitting your app or site on a daily basis, it’s also good to sit with your customers and understand how they’re using it in the real world and continue the loop of improving so you can more quickly hit your goals.

In the next post I’ll discuss How to Conduct a Usability Test on the cheap and if you’re interested in having someone take care of it for you, contact me and maybe we can work together.

What is Usability Testing?

By | implementation, product, user testing | One Comment

Over the next five posts I’m going to be outlining usability testing, specifically in regards to building websites, web applications or mobile apps. I’ve been working with a few different clients and partners recently and found myself explaining the ins and outs of why, when and how to run usability tests, which inspired a deeper dive and in turn this series of posts.

Usability testing is a process which allows you to validate your product’s ease of use with real users. Users are asked to complete tasks to see how they expect the product to work and uncovers any areas where they experience confusion or frustration.

A few example tasks are:

  • Use the website to purchase a shirt for yourself (for an ecommerce site)
  • Log-in to the web application and identify your balance for checking and savings accounts (for a banking web app)
  • Download the app from the app store and take a picture (for a mobile photography app)

During these tests users are encouraged to think aloud, explaining what they see and the decisions they’re making as they are progressing through the task.

Usability tests can be moderated or unmoderated and there are benefits to both and times in which you should use one over the other.

Moderated testing allows you to lead the user through a series of situations and if they get off track you can gently guide them back to the end goal at hand.

For my first round of testing, I like to use moderated testing in order to meet some of the users in person and have a conversation which can lead to other insights beyond the task at hand.

Unmoderated testing is typically cheaper and can be done using existing online tools. Some of these tools can help you find users for your test, which can be a sticking point to usability testing in general.

Another benefit of unmoderated testing is removing bias that the moderator can unintentionally cause in the user. Known as observer effect (Hawthorn effect), individuals modify or improve an aspect of their behavior in response to their awareness of being observed.

Usability testing can (and should) be used in a variety of ways during the product lifecycle. In the next post I’ll go over what kind of products can improve through usability testing as well as when to run usability tests.