What to Expect in the Usability Report

By | implementation, user testing | No Comments

From the first four posts in the series you now know the What, When, How, and Why of Usability Testing, centered specifically on User Testing.

When the test is done you will need to analyze the data from the test and write up your learnings. This is the usability report. If you’re working with a researcher like myself, then this is the deliverable that you will receive which breaks down the test and ultimately makes the recommendations for next steps. If you’re conducting the test yourself you should still take the time to write down your findings, even if it’s not quite as formal.

The summary should be a one page overview of the test that the stakeholder can review and have all the relevant information. This includes why you chose to perform the test, how many participants were tested and what persona(s) they fall into, the date(s) the test was performed on, and the key takeaways.

The introduction is a more in depth continuation of the summary explaining an overview of the decisions that brought you to the particulars of the usability test with details. An overview of usability testing and the steps you took prior to conducting the test.

Key Hypotheses
List the 1-3 hypotheses you made prior to executing the test. For example:

  • Users will be able to easily navigate from the welcome email into their account
  • Users will be able to buy a shirt after entering the home page
  • Users will be able to find and contact support

Now that you’ve conducted the test, what goals are you hoping to achieve?

  • Through a series of iterations, we will improve the comprehension of the data being presented by 80%.
  • Through a series of iterations, we will lower the user’s lostness score to below .2.

To read more about lostness score, read this Medium post by Shane Doyle.

Give an overview of your participants and include their demographic data and interview questions. If you have enough disparity add some pretty charts and graphs to show how they compare to one another.

Methodology & Scenarios
Some people seperate these into two sections, but I prefer to stick them together. Write up which testing methodology you used and then in a number list write up the scenarios you asked the users to walk through. You can copy them directly from the script you wrote for the test.

This section is broken down into four major parts:

  • Task Completion – A write up of each task and what the rate of success was for each user. Normally in the form of a table, followed up by detailed paragraphs.
  • Errors – any bugs that were found during the test.
  • Key Takeaways – Same as in your summary, the key learnings that you found in the test.
  • Notable quotes – one of my favorite parts. All those gems that your users spit out without even realizing how valuable they are. List them here. Mine tend to be a little longer than others I’ve seen because I’d rather have too many to pick from in the future than not enough.

The longest section by far. This is where you list all the problems you came across (beyond the key takeaways into the nitty gritty) and your recommendations about what to do about them. How I typically list them is:

  • State the problem – one sentence explaining the problem.  I also include a color-coded severity of the problem Low (green), Medium (orange) and High(red)
  • A screenshot for reference
  • Detail – an in-depth overview of the problem with reference to the participants issues
  • Recommendation – may include a simple wireframe to go along with the text recommendation
  • Additional comments (optional) – if there’s something beyond the recommendation that I feel needs to be addressed I’ll put it here

You repeat this until you’ve covered everything that has come up in the test.

This is usually 2-3 paragraphs analyzing the effectiveness of the test and suggestions for any next steps after the recommendations are implemented.

There you have it, a report to go along with all your hard work. As I recommended in the How portion of this series, UXPin has some great templates, including one for the report.

Next up: Pizza.

How to Run Your First Usability Test

By | implementation, user testing | No Comments

In my last two posts I wrote about the What and When of usability testing. Today I’ll be running through the How for first time testers.

There are a lot of different ways to study the usability of your product and depending on the situation you want to adjust your approach accordingly. The most useful place to start is with user testing and the great part is that there are some quick and easy steps so that  anyone can do it.

From a high level, you’re going to get users that are representative of real-life users, give them tasks to complete that real-life users would complete and watch what they do and how they do it.  Take notes, communicate what you see to your team and already you’ll have a better product. To do a slightly more formal study dive into the steps below.

Define Your Goal
First, what’s your goal? Just as you’ve defined the goal of your product, in turn you need to understand your goal of the test you’re going to perform before anything else. For example:

Can users find out about my consulting services?

Which method do users use to find a product in order to put it in the cart?

The important part in defining this is to create focus around the most important aspect of your product to test. The wider your testing lens (the more things you try to test) the more time a user has to begin learning your UI and creating skewed results. Remember the goal is to get a user’s initial reaction, not hoping they figure your UI out.

Create the Tasks
The next step is to define the tasks that you want the users to perform. What steps do you need them to take in order to achieve your goal?

Goal: Can users find out about my consulting services?
Task: Use the website to find out about any consulting services.  

Goal: Which method do users use to find a product in order to put it in the cart?
Task: Use the website to purchase a shirt for under $30.

Define Your Users
You need to understand who is going to be using your product. List out their primary characteristics so that when you go out to recruit subjects to help with your testing you can try to match as closely as possible to what a real-life user looks like. Some products have multiple users who will use the product differently. If that’s the case make sure to focus on the correct personas for the test you’re running. If multiple groups will need to achieve the goal you’ve laid out, make sure to get representatives from each set.

Write Your Plan
There are many good reasons to write your plan down, but the two most important are to make sure the stakeholders understand what you’re planning on doing and are on board and to have a clear layout throughout the testing.  

I still use this sample plan as a template when I conduct a user test:

And for the script part I always start with this:

Although over time mine has evolved into more of my voice and shifts based on what type of product I’m testing.

In addition UXPin has put together some great documents which you can find here, including a video consent form and a checklist for test day. They are a good starting point but definitely recommend making them your own.

Recruit Users
This is often viewed as one of the most difficult parts of the overall process. Finding the right kind of users (as you’ve defined above), scheduling times and preparing the documentation. For this, try to find users that are as close to real-life users as possible but don’t be super strict. Find friends, family or co-workers that fit the bill and ask them to do you a favor, especially if this is your first time through. Also, if you have existing users that’s how I’ve done the majority of my testing throughout my career.  They’re often more than happy to contribute to helping especially if they’re a fan.  

This article does a good job of laying out ways to recruit different types of users.

Conduct The Test
You can run the test face-to-face or over a screen sharing service. I’ve used in the past and in my most recent study used Go To Meeting, which worked well and was cheap.

Regardless of your proximity, address the user by name and thank them for their time. Make sure they are comfortable with the platform and then run through your script.

When it’s the user’s time to shine, say as little as possible and don’t answer any questions that might guide them to an solution that you’re hoping to find through the testing.  Let them know you’ll answer that question when the test is done or that you want to see how they work it out for themselves.

Once you’ve gone through all of the scenarios, ask any questions through what they saw or as follow up to the tasks they’ve just performed and make sure to give the user time to ask any questions of their own.

Write the Report
Using your notes and reviewing the video make an outline of the improvements that could be made to the site. Mark each change with a priority level and include both written and sketched suggestions so that you can add it to your next sprint or planning cycle. If it’s helpful you can even edit sections of the video to share with the engineering or designers so they can see the problem first hand.

I like to create a folder on Google Drive and include all the videos, the initial write-up and the final report along with a file linking to any stories or tickets I’ve created.  

As I said early, there are many other types of usability tests which I will go over in a future post, but for now you’re armed to go and run a user test and improve your product.

What else would you like to know about user testing or usability studies? Let me know in the comments.

Are you interested in having me run a usability study for you? Contact me.

What is Usability Testing?

By | implementation, product, user testing | One Comment

Over the next five posts I’m going to be outlining usability testing, specifically in regards to building websites, web applications or mobile apps. I’ve been working with a few different clients and partners recently and found myself explaining the ins and outs of why, when and how to run usability tests, which inspired a deeper dive and in turn this series of posts.

Usability testing is a process which allows you to validate your product’s ease of use with real users. Users are asked to complete tasks to see how they expect the product to work and uncovers any areas where they experience confusion or frustration.

A few example tasks are:

  • Use the website to purchase a shirt for yourself (for an ecommerce site)
  • Log-in to the web application and identify your balance for checking and savings accounts (for a banking web app)
  • Download the app from the app store and take a picture (for a mobile photography app)

During these tests users are encouraged to think aloud, explaining what they see and the decisions they’re making as they are progressing through the task.

Usability tests can be moderated or unmoderated and there are benefits to both and times in which you should use one over the other.

Moderated testing allows you to lead the user through a series of situations and if they get off track you can gently guide them back to the end goal at hand.

For my first round of testing, I like to use moderated testing in order to meet some of the users in person and have a conversation which can lead to other insights beyond the task at hand.

Unmoderated testing is typically cheaper and can be done using existing online tools. Some of these tools can help you find users for your test, which can be a sticking point to usability testing in general.

Another benefit of unmoderated testing is removing bias that the moderator can unintentionally cause in the user. Known as observer effect (Hawthorn effect), individuals modify or improve an aspect of their behavior in response to their awareness of being observed.

Usability testing can (and should) be used in a variety of ways during the product lifecycle. In the next post I’ll go over what kind of products can improve through usability testing as well as when to run usability tests.

How To Define Your Product Vision

By | implementation, product, starting, vision | No Comments

All the way back on Day 2 of my 31 days of blog posts, I asserted that to build great products you needed to Know Your Vision, inside and out. Understand your why and shout it from the rooftops, or at least be able to spit out an elevator pitch.

“People don’t buy what you do, they buy why you do it.” – Simon Sinek, Start With Why

I’ll expand that for the product managers out there to say that you can’t expect your team to blindly support what you’re building, but they will bend over backwards if they believe in why they are building it.

One of the most important responsibilities for a product manager is to inspire and lead their teams towards a common goal. No matter the size of the company or the size of your product, you need an overarching goal in order to create a strategy on how to get there. On top of that, your vision is the thing you point to in moments of doubt or to justify changes in strategy.

This goes beyond product and engineering. It should also guide marketing, sales and the team that will have to support your product before, during and after launch.

The way that I like to think about it is that you’re painting a version of the future that is better because of what you’re building. A future where what you’re building is delivering value to both your business and your customer and they’ve got big smiles on their faces.

For SpaceX, their vision is enabling human life on Mars.

Talk about a vision. I’m going to go out on a limb and guess that the product you’re building is not quite as grandiose in scope, but in this case size doesn’t matter. You should have the same passion and ambition behind your vision.

Before you can truly define your product vision you have to understand your product. As the product manager, that’s your job. You need to understand all aspects of your product, beyond the features to include:

  • Key Product Goals
  • Target Customers
  • Competition
  • Differentiation

One way to go about creating a vision is to follow the template from Geoffrey Moore’s book Crossing the Chasm.  All you have to do is fill in the blanks:

  • For (target customer)
  • Who (statement of the need or opportunity)
  • (Product name) is a (product category)
  • That provides (key benefit)
  • Unlike (the product alternatives)
  • Our product (statement of primary differentiation)

You’ll end up with a vision statement like this one I made up for
For the bill-paying member of the family who also manages the budget and is tired of tracking multiple accounts in order to have a clear financial overview. is a web-based program that automatically ties in to all financial institutions and automatically updates in real time. It is optimized specifically for the everyman budgeter and is free to use.

It’s a fine vision statement but, it doesn’t get me jazzed and quite frankly it’s way to long. You want to shoot for short and sweet. Could I use it to point to and support my product decisions.  Sure, but getting engineers and salesmen to recollect it and follow it is a stretch. Plus, it’s just not cool.

It’s a good starting point, and in fact is a great exercise to go through with your team and make sure that the foundation of the vision is solid throughout your organization. As the product owner it is your responsibility to create the vision statement, but along the way you should take the time to include everyone else that has a stake in the product and get their input. Involving your partners early and often will ensure that everyone is on the same page even before you distribute the clean, concise, dare I say, sexy final version.

Now, take that statement and break it down to it’s most essential pieces. What’s the heart of the product that you’re trying to create for your target customers. How will their lives be better and different in the future? Break that out and craft a statement or two.  Then take another pass and see if it gets you excited.

“If you are working on something exciting that you really care about, you don’t have to be pushed.  The vision pulls you.” – Steve Jobs

Is what you have now pulling you? Is it ambitious? Will it engage the troops when it’s time to kick it into high gear? Is it short and sweet? And most importantly, does it make sense?

If not, rework it until it does.  When you’ve got a vision that excites you and paints that better version of the future, then communicate it throughout your organization and ensure that everyone is inline, from the C-level down to your entry-level engineers.

With this, you’ve set the direction and everyone is moving together towards the higher purpose and a full understanding of your why, which ultimately makes your team more of a team.

Here’s a few examples that I like:

To build a place where people can come to find and discover anything they might want to buy online.

To inspire and implement solutions to the environmental crisis.

To create a better everyday life for the many people.

Toys ‘R Us:
Put joy in kids’ hearts and a smile on parent’s faces.

Final thought:
One thing I liked about SpaceX’s mission is that before stating the goal they also said this “SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not.”

That is heart and passion and raw, real, truth.  So when you’re looking to create your vision, think about SpaceX and let it be a little raw.