When Should I Do Usability Testing?

By | product, strategy, user testing | No Comments

In the last post I explained what a usability study is. If you haven’t read that, check it out here.  

Today I wanted to focus on when to do a usability study, but before we dive into that let me first address what types of products benefit from a usability study.

When I talk about usability studies I’m typically focused on tech products which I group into websites, web applications, native applications or mobile apps. I do believe there is benefit in bringing in the user for other types of products during the entire lifecycle and getting feedback along the way as you design and build, whether that be something physical, like the next great widget or something like a book, which Ash Maurya lays out in great detail in his book Running Lean (for that very book).

Bringing it back to tech products my answer would be that they all benefit from usability studies. I like to focus on the low and high end of complexity as use cases and letting the middle sort itself out, so in that spirit let’s break it down:

Low complexity
A portfolio website to show off your artwork and increase consulting work. Surely you don’t need to do any usability testing. It’s straightforward, here’s some beautiful design work and here’s how to get in touch with you. Boom, done. My counter argument to that line of thinking is, what if you could increase the amount of consulting work you get by 10% with a few hours of work? How about 50%? By putting the site in front of people with the task of hiring a design consultant you could discover that they want to fill out a form rather than call or email you. That small change could increase your rate of return.

High complexity
An advertising platform that allows advertisers and publishers to manage their accounts in real time. This would be a web application with a mobile component that allows  both external and internal users different levels of access. With this type of product you would have many levels of usability testing. From a persona level you have advertisers and publishers, all of which are probably not the same and would need to be broken down into sub groups. Even within those personas it’s often the case that users will need different levels of access, such as the Sales Executive and the Ad Buyer. That’s without yet considering how the internal account managers will need to impersonate users and have their own switches and levers.  

Putting these products in front of users is crucial to saving time and money as you’re building and maximizing profits after it hits the market.  

But when do you put it in front of users? When is the right time to do usability testing?

The short answer is, throughout your entire process, from discovery to design, through development and even after launch.

In Tom Chi’s talk on Rapid Iteration (which I posted back in December) he says, “For everyone who builds products, the real medium is human behaviour in the real world, and your purpose should be…to see that human behaviour modified by what you build.”

The longer answer is that the more usability testing you can achieve during your product creation, the better. In Tom Chi’s talk he mentions a feedback loop of only a few days, constantly giving feedback to the design and engineering teams. That is a high bar to reach for indeed.

Instead, breaking the project down into Discovery, Design, Development and Launch we can start to identify the points at which usability tests can be effective, even if we aren’t on a bi-weekly cycle with our feedback loop.

Product discovery starts with understanding what you’re hoping to achieve. See my post on setting your product vision. Involving users at this stage typically means conducting user interviews or nothing at all. One thing you could do as you’re qualifying your intuition is put users in front of similar products and gain insight from how they interact.

During the design phase I encourage iterative design and adding in parallel design if you have the time. Iterative design is the process of taking user feedback on a design and updating it with marked usability improvement with each iteration that you do.  Parallel design starts with multiple concepts and takes the best from all three to end up at 1 design to start the iterative process from.

You want to work through as many cycles as you can, continually gaining insight from users, before a line of code is ever written. In this great article on the Nielsen Norman Group website it goes through the details of each of these projects and state “there’s no one perfect user interface design, and you can’t get good usability by simply shipping your one best idea. You have to try (and test) multiple design ideas.”

During this phase UX, design and engineering should be talking every day about progress of the project and UX should be thinking about what they’re going to test on a weekly basis. In the video from Tom Chi above his team does it twice a week. Do what you can, if you can only do it once every two weeks it’s better than not at all. By taking what engineering is building and putting it in front of users and finding the flaws (and really awesome stuff) as you’re building it will save you from huge rewrites and updates after launch which saves your team huge amounts of time and money.

As I wrote about a few days ago, when you launch you’re not finished, keep pushing towards your goal that you set up at the beginning of the project and while now you will have quantitative data coming in through the many users you have hitting your app or site on a daily basis, it’s also good to sit with your customers and understand how they’re using it in the real world and continue the loop of improving so you can more quickly hit your goals.

In the next post I’ll discuss How to Conduct a Usability Test on the cheap and if you’re interested in having someone take care of it for you, contact me and maybe we can work together.

What is Usability Testing?

By | implementation, product, user testing | One Comment

Over the next five posts I’m going to be outlining usability testing, specifically in regards to building websites, web applications or mobile apps. I’ve been working with a few different clients and partners recently and found myself explaining the ins and outs of why, when and how to run usability tests, which inspired a deeper dive and in turn this series of posts.

Usability testing is a process which allows you to validate your product’s ease of use with real users. Users are asked to complete tasks to see how they expect the product to work and uncovers any areas where they experience confusion or frustration.

A few example tasks are:

  • Use the website to purchase a shirt for yourself (for an ecommerce site)
  • Log-in to the web application and identify your balance for checking and savings accounts (for a banking web app)
  • Download the app from the app store and take a picture (for a mobile photography app)

During these tests users are encouraged to think aloud, explaining what they see and the decisions they’re making as they are progressing through the task.

Usability tests can be moderated or unmoderated and there are benefits to both and times in which you should use one over the other.

Moderated testing allows you to lead the user through a series of situations and if they get off track you can gently guide them back to the end goal at hand.

For my first round of testing, I like to use moderated testing in order to meet some of the users in person and have a conversation which can lead to other insights beyond the task at hand.

Unmoderated testing is typically cheaper and can be done using existing online tools. Some of these tools can help you find users for your test, which can be a sticking point to usability testing in general.

Another benefit of unmoderated testing is removing bias that the moderator can unintentionally cause in the user. Known as observer effect (Hawthorn effect), individuals modify or improve an aspect of their behavior in response to their awareness of being observed.

Usability testing can (and should) be used in a variety of ways during the product lifecycle. In the next post I’ll go over what kind of products can improve through usability testing as well as when to run usability tests.

Understanding Where The Finish Line Really Is

By | product, vision | No Comments

I’ve worked on far too many projects that were treated as if the feature launch were the end of the project. Hooray, the feature is live and now we can get onto the next thing on our list. It’s the way it was done for a long time and truthfully it’s the way a lot of companies still build products today.

Just like many things, the first step towards fixing a problem is admitting you have a problem.

Over the next few days I’m going to be writing a series on usability testing, the what, when and how of it all, but as I was planning out those posts it got me to thinking about this issue that I know is still plaguing companies. Teams are agile and scrumming and running sprints, all towards this illusion of being done.  But when do you really know that you’re done?

The definition I use for done is when the customer is happy and the most important metric I’ve been tracking towards has been hit. Those two things tell me I’m done, not a feature release date. I like to think of the feature release date as a baton handoff in a relay race. If all has gone well this is the second such handoff. The first such handoff should happen after testing and experimenting during the design phase as I shift over to the development phase with the engineering team. That is not to say that all teams shouldn’t be involved during all phases of the process, in fact UX, design and engineering should be communicating all along the way, rather it’s just the phase of the project you’re in.

After the release of a feature you should be checking your data to make sure that it’s in line with the hypothesis you made prior to launch and allocating proper resources in order to make changes necessary in order to move the needle towards your goal. The resources used during this step should be greatly diminished from the major push which was just completed, but the problem is that many companies allocate no more time on the completed feature. That is why there is no true learning and features that are not contributing the way they should to the overall product.

It could take you months to drive the numbers in the way you need them to go, but keep making small tweaks and changes and follow the data. Once you feel like the feature is performing as expected and adding the value you originally intended, then it is time to move it over to maintenance mode.

Maintenance mode is a point at which further investment into improving your numbers will be a waste of resources, delivering diminishing returns, so at this point you set a goal (maintenance goal) in order to verify that the feature is staying the course and not dropping, but with little support.

Would you like to know more on how to have these conversations and begin to create a culture of learning? Let me know in the comments below.

Product Lessons from Mark Zukerberg’s Home AI Challenge

By | product, starting, strategy | No Comments

A few days ago Mark Zukerberg posted to his Facebook page about his experience working on building a simple AI to run his home, which he aptly named Jarvis after Tony Stark’s AI from Iron Man. And the same day Fast Company wrote an article on it. You may have heard about it.  

I also discovered that Mark Zukerberg sets a yearly challenge for himself. So, in addition to being CEO of Facebook and having a family he makes it a priority to continue to learn and grow. 

Throughout his update, there were lessons on the right way to build products. He used creative problem solving, iteration and existing software rather than building from scratch to get to his vision as quickly as possible.

From the very onset of the project he faced challenges because there is currently no common API for the home appliances. As home automation AI becomes more prevalent (and it will happen quickly) this is gap that will need to be filled.

He had particular trouble finding a toaster that would allow you to push the bread down without having it turned on. “I ended up finding an old toaster from the 1950s and rigging it up with a connected switch.”

This is an example of taking a problem and turning it on it’s head. There were probably other more complicated solutions, like reengineering a present day toaster, but he took the problem and broke it down to its smallest parts and attacked it from a new angle. This type of creative problem solving is necessary as we navigate building products.

Another hurdle was allowing Jarvis to understand voice commands. As we should when building any new product he took an iterative approach by first communicating with Jarvis via text message and later using that as a springboard to allow for voice commands.

I was particularly interested by his discovery that he often prefers to communicate with Jarvis via text rather than using voice commands. And based on this observation he determined that as we build out AI, we need to recognize that they will also need messaging interfaces in addition to voice.

Building iteratively and real world experimentation and discovery at work.  

In order to create the messaging interface he could have built an app from scratch, but chose instead to build a Messenger bot.  Why? Because it was easier. There are open source APIs to build these bots, which allowed for a strong base to start with. When building towards MVP it’s important to move as quickly as possible and often that can be done through retrofitting your ideas into existing solutions, even if it’s not as sexy as you imagine it to be when you’re dreaming of the final version.  

Beyond his iterative approach, creative problem solving, and building on top of existing work it’s important to acknowledge the very fact that he challenged himself to do this. If he had never put pen to paper and challenged himself at the beginning of the year there’s little chance that he would now have a kick ass home AI, not to mention the knowledge gained, including learnings he wasn’t expecting at the beginning, which is common in taking on big challenges. You just don’t know what you don’t know yet.

As I’ve been saying from my very first post. Start. 

Then, make small iterative progress towards whatever it is you’re looking to do. Set a goal and break it down into bite sized chunks and then break it down some more.  Make it so easy that you’d be hard pressed not to make a little progress every day.  

From beginning to end it took Zuckerberg a little over 100 hours to get to the current version of Jarvis.  Think about what you could do in 100 hours over the course of this coming year.  

What challenge do you want to tackle in 2017? Let me know in the comments below.

How To Define Your Product Vision

By | implementation, product, starting, vision | No Comments

All the way back on Day 2 of my 31 days of blog posts, I asserted that to build great products you needed to Know Your Vision, inside and out. Understand your why and shout it from the rooftops, or at least be able to spit out an elevator pitch.

“People don’t buy what you do, they buy why you do it.” – Simon Sinek, Start With Why

I’ll expand that for the product managers out there to say that you can’t expect your team to blindly support what you’re building, but they will bend over backwards if they believe in why they are building it.

One of the most important responsibilities for a product manager is to inspire and lead their teams towards a common goal. No matter the size of the company or the size of your product, you need an overarching goal in order to create a strategy on how to get there. On top of that, your vision is the thing you point to in moments of doubt or to justify changes in strategy.

This goes beyond product and engineering. It should also guide marketing, sales and the team that will have to support your product before, during and after launch.

The way that I like to think about it is that you’re painting a version of the future that is better because of what you’re building. A future where what you’re building is delivering value to both your business and your customer and they’ve got big smiles on their faces.

For SpaceX, their vision is enabling human life on Mars.

Talk about a vision. I’m going to go out on a limb and guess that the product you’re building is not quite as grandiose in scope, but in this case size doesn’t matter. You should have the same passion and ambition behind your vision.

Before you can truly define your product vision you have to understand your product. As the product manager, that’s your job. You need to understand all aspects of your product, beyond the features to include:

  • Key Product Goals
  • Target Customers
  • Competition
  • Differentiation

One way to go about creating a vision is to follow the template from Geoffrey Moore’s book Crossing the Chasm.  All you have to do is fill in the blanks:

  • For (target customer)
  • Who (statement of the need or opportunity)
  • (Product name) is a (product category)
  • That provides (key benefit)
  • Unlike (the product alternatives)
  • Our product (statement of primary differentiation)

You’ll end up with a vision statement like this one I made up for
For the bill-paying member of the family who also manages the budget and is tired of tracking multiple accounts in order to have a clear financial overview. is a web-based program that automatically ties in to all financial institutions and automatically updates in real time. It is optimized specifically for the everyman budgeter and is free to use.

It’s a fine vision statement but, it doesn’t get me jazzed and quite frankly it’s way to long. You want to shoot for short and sweet. Could I use it to point to and support my product decisions.  Sure, but getting engineers and salesmen to recollect it and follow it is a stretch. Plus, it’s just not cool.

It’s a good starting point, and in fact is a great exercise to go through with your team and make sure that the foundation of the vision is solid throughout your organization. As the product owner it is your responsibility to create the vision statement, but along the way you should take the time to include everyone else that has a stake in the product and get their input. Involving your partners early and often will ensure that everyone is on the same page even before you distribute the clean, concise, dare I say, sexy final version.

Now, take that statement and break it down to it’s most essential pieces. What’s the heart of the product that you’re trying to create for your target customers. How will their lives be better and different in the future? Break that out and craft a statement or two.  Then take another pass and see if it gets you excited.

“If you are working on something exciting that you really care about, you don’t have to be pushed.  The vision pulls you.” – Steve Jobs

Is what you have now pulling you? Is it ambitious? Will it engage the troops when it’s time to kick it into high gear? Is it short and sweet? And most importantly, does it make sense?

If not, rework it until it does.  When you’ve got a vision that excites you and paints that better version of the future, then communicate it throughout your organization and ensure that everyone is inline, from the C-level down to your entry-level engineers.

With this, you’ve set the direction and everyone is moving together towards the higher purpose and a full understanding of your why, which ultimately makes your team more of a team.

Here’s a few examples that I like:

To build a place where people can come to find and discover anything they might want to buy online.

To inspire and implement solutions to the environmental crisis.

To create a better everyday life for the many people.

Toys ‘R Us:
Put joy in kids’ hearts and a smile on parent’s faces.

Final thought:
One thing I liked about SpaceX’s mission is that before stating the goal they also said this “SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not.”

That is heart and passion and raw, real, truth.  So when you’re looking to create your vision, think about SpaceX and let it be a little raw.