How Heuristic Analysis Improves eCommerce UX

Everyone’s a critic, right?

Sometimes that’s not a bad thing, though. By gathering the critiques of observers — either from experts or the internal team — Heuristic Analysis can isolate a lot of usability problems before they ever become problems. While not a substitute for proper user testing, it can certainly help with the heavy lifting, and at a fraction of the cost and effort.

Just look at these statistics compiled by usability authority Jeff Sauro:

A typical Heuristic Analysis will uncover between 30-50% of usability problems. In Sauro’s sample, the analysis revealed all the top ten most common usability issues, and 75% of the top 20. It may uncover (less frequent) issues that user testing misses.

Successful sites need user testing, but given the resources needed for testing, an early Heuristic Analysis will relieve some of the burden. They make user tests less intensive (cheaper) and increase the likelihood of finding all the major problems.

Naturally, these advantages would interest most designers, so here’s a quick reference guide on everything you need to know to get started.

Benefits & Drawbacks of Heuristic Analysis

What does Heuristic Analysis bring to the table that user testing doesn’t? What do you have to be careful of? Take a look below.

Benefits

Focuses User Tests. The main benefit of Heuristic Analysis is that it reveals minor issues early on so you can correct them in the beginning stages. After removing the “low-hanging fruit,” as Ari Weissman calls it, your later user tests focus exclusively on the major issues. With fewer issues to worry about, participants in a user test will have more time to hone in on the greater problems.

Cost. The price tag on Heuristic Analysis is highly scalable. You can spring for the best experts in your field, or your team can take a few extra hours to conduct one themselves. Of course the results will vary, but, as a way to cut down on later user testing, a small inexpensive heuristic analysis can ultimately save you money. In fact, the Nielsen Norman Group lists Heuristic Analysis as one of the three techniques in their “discount usability engineering” method.

Prioritizes Known Issues. As an aid to user testing, Heuristic Analysis gives you the preliminary data necessary to better organize and prioritize the issues to test for. This early usability data can even go as far as determining which types of tests are needed to target specific problems.

Show Your Rank Against Competitors. Because Heuristic Analysis can quantify abstract design concepts, it’s easier to compare how your site fares against competitors. The data from the analysis can also be converted into charts for more effective comparisons, as we explain below.

Speed. Heuristic analysis can be done quickly and with only slight preparation, even while the user tests are being planned.

Drawbacks

Not User-Based. The main problem with Heuristic Analysis (and the reason it can’t replace user testing) is that it’s not specific to the user. It’s only the observations (“guesswork,” as critics call it) of outsider groups, not the everyday users of the site. Even experts are confined to their own expertise, so the only true authority are the people who will actually use it.

Limited Perspective. While the heuristics you choose to access help focus the analysis, they also restrict it: the review only measures those metrics, and nothing else. Moreover, the scope of analysis is limited to whom is chosen to evaluate. These reasons are why Heuristic Analysis only covers a portion of the usability issues and not the whole.

Process

As stated above, Heuristic Analysis is both quick and cost-effective. All you need to start is a working prototype or the completed site. In a pinch, you can use screenshots or mockups, although these aren’t accurate depictions of usability and will compromise your results.

1. Choose the Heuristics. First, you need to narrow in on what exactly you’re analyzing. These could be a standard list of usability metrics, or a list of your own design. In the next section, we review three different sets of heuristics to choose from.

For a more controlled analysis, you can create tasks for your evaluators; although a broad page-by-page review is also effective.

2. Choose and Brief the Evaluators. Depending on your budget and restraints, you can hire a professional team of evaluators or have your internal team do it. There are two basic requirements for the evaluators:

Between 3-5 evaluators. Sauro’s research concluded that between 3-5 evaluators is best. Under three, the evaluators fail to catch enough issues. Over five, the results plateau, so it’s a bit overkill.

Expertise. The evaluators should have a background in usability or the site’s field — but preferably both.

Make sure you brief the evaluators on which heuristics they are using before they begin.

3. Conduct the Heuristic Analysis. Have the evaluators review your site or prototype. While the procedure tends to be up to the evaluator themself, the NNG recommends at least two run-throughs of the site over a time period of 1-2 hours.

Additionally, it’s helpful to conduct an analysis of your competitors’ sites as well. This shows you how you rate in your field as a whole, plus reveals weaknesses in your rivals that you can leverage, or in yourself that you can improve.

The evaluators should mark their results on a scorecard. Feel free to use this template card from UX for the Masses, although the heuristics have been pre-selected.

4. Analyze the Results. Collect all the scorecards and deconstruct the data (if the review agency doesn’t do this for you). You may find it helpful to create graphs as visual aids, especially if you’ve evaluated your competitors as well. Michael Hawley recommends using a spider web diagram. Aside from being clear visual aids, these graphs make comparisons easier because they can all be overlapped on top of each other.

In a spider web graph, each heuristic is represented by a leg extending out from the center. During the evaluation, each one is given a numeric score, and the scores are then graphed and connected. You can draw each graph on transparent paper and then overlap them, or simply use different colors on the same graph.

Whether comparing different pages or different sites, this method clearly displays the strengths and weaknesses of each source, according to your chosen heuristics.

The Importance of Choosing Your Heuristics

The results of the analysis are determined by the heuristics you choose. A beauty pageant wouldn’t be effective if the judges’ metrics were things like weightlifting or how long they can hold their breath under water.

Let’s review some of the most popular options.

Nielsen Norman Group’s Heuristics

The most traditional heuristics list is also the most thorough. Developed by the company that helped bring Heuristic Analysis to the forefront, NNG’s list remains as useful today as it was when it was first compiled more than 25 years ago.

Visibility of System Status — The user should have enough feedback to know what’s going on, and within an appropriate amount of time.

Match Between System and Real World — The site follows real-life conventions — especially language — rather than system-specific communication.

User Control and Freedom — The user can move about the site freely, especially if they make a mistake and must backtrack.

Consistency and Standards — The site maintains consistency with functionality and terminology.

Error Prevention — Common errors are preemptively eliminated, and confirmation and undo options are in place for other potential errors.

Recognition Rather than Recall — Relieve the memory strain of users by having options visible or easily retrievable.

Flexibility and Efficiency of Use — The user experience is streamlined to minimize user effort.

Aesthetic and Minimalist Design — Cut out all unnecessary content, especially for dialogue windows.

Help Users Recognize, Diagnose, and Recover from Errors — Error messages are expressed plainly and suggest a solution.

Help and Documentation — While not always necessary, help documentation should be easy to find and clearly answer users’ questions.

Peep Laja’s Heuristics

The founder of ConversionXL Peep Laja uses more simplified umbrella categories for a less intensive Heuristic Analysis. Each heuristic represents a collection of issues, so make sure to give each the attention it deserves.

Clarity — The site’s communication across a broad range, including content, navigation, and the value of the site itself.

Friction — Make sure there are no redundancies and that all obvious concerns (like security and privacy) are addressed.

Distraction — Screen elements don’t have to compete for attention, and the user can find what they want without stress.

Urgency — The visual hierarchy is as it should be, with the most actionable elements prioritized over the lesser ones.

Traffic Sources and Scent — The page delivers what the user expects, depending on the source.

Pacing (Buying Phases) — How well the page fits into the overall journey, which is especially important for eCommerce pages, where content much match the step-by-step sales funnel.

ACHI 2014 eCommerce Heuristics

The Seventh International Conference on Advances in Computer-Human Interactions (ACHI 2014) presented this academic paper on the 64 most useful heuristics for eCommerce sites, based on the results of three previous studies. The beauty of this list is that it’s tailored just for eCommerce.

The heuristics are divided into six categories that correspond to five phases of the Buying Decision Process, and one overview category.

Need Recognition and Problem Awareness Information Search Purchase Decision Making Transaction Post-Sales Behaviour Factors that Affect the UX during the Whole Purchase Process

We highly recommend that eCommerce sites read this paper, at least for inspiration. While 64 heuristics might be too difficult to test, you can still pick a few pertinent ones to add to your list.

Sample Heuristic Analysis

Heuristic Analyses can be as detailed or abbreviated as you wish, can review as many or as few heuristics as you need, and can be evaluated with a quantitative score or a qualitative description as you prefer.

To get you started, this single evaluator’s Heuristic Analysis of Hennepin County Library website can shed some light on what the end goal of this practice is.

The entire report is sectioned off by the individual heuristic. The project chose to use the NNG’s traditional 10 heuristics, with a textual report of the performance for each.

Notice the inclusion of screenshots and arrow markers to back up the reviewer’s points. These visual aids demonstrate the points to those less familiar with the interface and may refresh the evaluator's memory later. In general, it’s good to have a document as complete as possible — years later, you may need to research previous heuristic reviews.

Take some time to browse this document and get an understanding of how a finished Heuristic Analysis feels.

Takeaway

Don’t forget that Heuristic Analyses are just a secondary aid for improving the design process, but they are not an end in themselves. No matter how skilled the evaluator is, there’s no substitute for actual user data.

That said, Heuristic Analysis is great at “clearing a path” for user testing. Because user testing is crucial — and also resource-consuming — you want to optimize the results while minimizing the costs.

That’s what Heuristic Analysis is for: taking care of the smaller issues so user testing can devote all its focus on the bigger issues. If a user is complaining about a minor issue like an inconsistency in the navigation menu, they might miss a more telltale issue, like a way to make the navigation menu more intuitive.

Think of Heuristic Analysis as the little, seemingly insignificant tug-boat that pulls the enormous ocean-liner to where it needs to go.

About the author

Matt Ellis is a design writer and online content strategist. He's been writing professionally for over 10 years, with the last 5 focusing exclusively on online marketing through content creation and distribution. For his professional work, visit his website Matt Ellis.