Measuring UX with HEART

One of the most important steps in developing any digital project is defining what success means is. Because we have the possibility to measure virtually everything, it’s easy to get caught in the trap of measuring all the wrong things. We often throw these numbers around and, of course, the bigger the number is, the more impressive it is. We tend to do this with almost everything nowadays! Page views, email subscribers, instagram followers, funding raised, revenue, and even numbers of employees. Big is better, right? Don’t get me wrong, in some cases big is better - like profit, but in some cases it’s just vanity metrics.

The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.

When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.Eric Ries on Tim Ferriss

With all of the tools available showing us all the charts, how do we know which are useful and which are just vanity metrics? Hint: the answer is right there in the question — “u-s-e-f-u-l.” Metrics that help us make decisions are useful, metrics that can’t be used to make actionable steps are vanity metrics.

It all comes down to one thing: does the metric help you make decisions? When you see the metric, do you know what you need to do? If you don’t, you’re probably looking at a vanity metric. Vanity metrics are all those data points that make us feel good if they go up but don’t help us make decisions.Neil Patel - Metrics, Metrics On The Wall, Who’s The Vainest Of Them All?

Measuring UX

While other areas have it easier to measure their progress in numbers, UX is still tougher to tackle. Because UX is so difficult to measure, we still struggle to define our value to organizations who could benefit from our work.

Business metrics are often tied directly to dollars and cents, so it’s a metric almost everyone understands. Improved sales and reduced costs equal profit and everyone loves money. If it’s a business that relies on technology they already have tons of stuff they measure. Reduced server loads, faster software, and less latency are all things that while not everyone in the public truly understand them, most understand them as a good thing.

But UX, and design in general, is harder to tackle. One way to measure its effect is through the HEART framework, designed by Kerry Rodden, Hilary Hutchinson, and Xin Fu from Google’s research team.

To make this work in practice it’s important to use the right metrics. Basic traffic metrics (like overall page views or number of unique users) are easy to track and give a good baseline on how your site is doing, but they are often not very useful for evaluating the impact of UX changes. This is because they are very general, and usually don’t relate directly to either the quality of the user experience or the goals of your project — it’s hard to make them actionable.How to choose the right UX metrics for your product

The framework is a kind of UX metrics scorecard that’s broken down into 5 factors:

  • Happiness: How do users feel about your product? Happiness is typically measured by user satisfaction surveys, app ratings and reviews, and net promoter score.
  • Engagement: How often are people coming back to use the product? Level of user involvement, typically measured via behavioral proxies such as frequency, intensity, or depth of interaction over some time period. Examples might include the number of visits per user per week or the number of photos uploaded per user per day.
  • Adoption: How many people successfully complete the on-boarding process and become regular users? Adoption is measured by number of new users over a period of time or percentage of customers using a new feature.
  • Retention: The rate at which existing users are returning. You can measure how many of the active users from a given time period are still present in some later time period. As a product owner, you may be more interested in failure to retain, commonly known as “churn.”
  • Task success: Can your users achieve their goal or task quickly and easily? Task success is measured by factors like efficiency (how long it takes users to complete the task), effectiveness (percent of tasks completed), and error rate.

Using the HEART framework gives you a better understanding of the user and their relationship with the product. The nice perk of this is that it can be applied to a single feature in your app or to your whole product. It gives you the option to measure just what you need to at any moment.

Informed by data, driven by empathy

The other day, I published this quote from Data-Driven Design is Killing Our Instincts that’s stuck with me.

We’re told all design decisions must be validated by user feedback or business success metrics. Analytics are measuring the design effectiveness of every tweak and change we make. If it can’t be proven to work in a prototype, A/B test, or MVP, it’s not worth trying at all. 

In this cutthroat world of data-driven design, we’re starting to lose sight of something we once cherished: the designer’s instinct. “Trusting your gut” now means “lazy, entitled designer.” When we can ask users what they want directly, there’s no room for instinct and guesswork. 

Or is there?

As we have access to all of this data, and the general consensus is that data should lead the way, I’m confused by what the future role of a designer is. What I truly like about the HEART framework is that while the output of it is informed data, it’s all done through the eye and mind of a human. Just like we need to be careful to not measure everything just because we can, we need to remember that in order to really understand human’s behaviors, we need to talk with them, not just look at charts. Charts are great for an overview, but human insight is the best way to understand the underlying reason for the chart’s direction.

“Data-driven” is all the rage at the moment, everyone wants a slice of the “big data” cake. Data scientists are the new rock stars, replacing the JavaScript and Front-end gurus and ninjas from a few years back. My problem with trends like these is that they cause the so called “tunnel-vision”. Thing x is a trend right now and we should do that too because,… you know… everyone’s doing it. Some companies have already started to realise that data alone can’t answer all the questions they need answered. Booking.com put “Informed by data, driven by empathy” in their design guidelines.Measuring and Quantifying User Experience

How do you get from the HEART categories to metrics you can actually implement and track? Unfortunately, there’s no off-the-shelf “HEART dashboard” that will magically do this for you. The most useful metrics are likely to be specific to your particular product or project. Personally, I like to start bigger projects with a UX Strategy. This UX Strategy should define what the most important tasks and metrics for your product is, and consequentially, what failure looks like. How might success or failure in the goals actually manifest itself in user behavior or attitudes? For YouTube, an engagement signal might be the number of videos users watch, but an even better one could be the amount of time they spend watching those videos. A failure category for YouTube Search might be entering a query, but not clicking on any of the results.

Even fashion brands are using data to design products that will meet customers demands better:

“You never design by data, but the data provides a compass as you’re navigating a hunch.”Analytics are reshaping fashion’s old-school instincts

So while we as designers might not get the out-of-the-box dashboard with all the charts ready for us to act on, there are tools and techniques that we can use to get the information needed for us to act upon to get a confident start. Let me know if you need help getting started!

Please share:TwitterLinkedInFacebook
Get more writing like this

Sign up and get new writing, just like this, every other two weeks. Unsubscribe any time (I'm not a dickhead).

Books

User Experiences that Matter (2016)
Mastering Freelance (2017)

If You're Getting Started in UX

What's a 'User Experience' Anyways?
How Do You Learn UX?
Working as a UX Designer

Next Steps in UX

Working as a UX Lead
Defining a UX Strategy
Writing as Part of the UX Process

Thought-pieces

AI Ethics - A New Skill for UX-Designers
Designer Ethics & The Moral Implications of our Apps
The Future of the UX-Designer
Voice Input’s Effect on Social Norms

The Work We Do

Chasing Growth
New Tools Don’t Always Equal Productivity
Why Designers Need to Write
The Tools I Use to Run My Business

Featured Writing & Interviews Elsewhere

Q&A With Anton Sten, Author of User Experiences that Matter - Adobe
What the F*#!ck is a UX Designer anyway - Working not Working
It’s Time for a Code of Ethics for Designers - Medium Modus
The Art of Going Freelance - .Net Magazine
It Takes Time - Being Freelance episode 100

From My Newsletter

UX of Email Newsletters

Working as a UX-lead

2018 in review

What’s my location?

I’m taking a break

Stay humble, stay eager

Back to Work!

Vanity Metrics

The Future of Retail

2017 review

What do you do?

Carpe Diem UX-designers

What´s Good Design?

Chasing Growth

A Redesign

Is Less More?

Why Simple is Hard

Pricing It Perfectly

Trusting Your Gut

Built to last

An Eye on the Future

UX Design explained

Bite-sized Posts

The Enemy

The Hot Potato Process

Leave the Phone at Home

Delight Comes Last

The Cost of Lies

Big Mood Machine

Simplicity is a war

The next iPhone

Leadership or management

Everyone should own a dog

Silence is gold

Cameras that understand

Humans, not users

Keeping AI Honest

Right to privacy

The State of UX in 2019

Why scrap scrappy?

Organized for browsing

The iPhone Franchise

Why Small Teams Win

The Bullshit Web

Just keep at it

Let them eat cake

Netflix Culture

Skype

Unfoundered

Tech is not Neutral

Productivity

Whose risk?

Why Small Teams Win

Phone Bored

Karim Rashid

Dieter Rams

Bleeding Out

Fake News is spam

Conversational Design

Dropbox

Bye bye Facebook

Cuba

The seat at the table

Givenchy

Love letters to trees

Pricing Philisophy

Specialize

Personas

Make me think

Hawaii Missile Alert

Why Design Systems fail

What You Build

Checkout for Winners

Living a Testing Culture

Creative Class

How To Predict Your Future

Github

Medium

Design quotes

Enough

Why?

ARKit

Designing for Mobile

Failure, Reflect, Renew

Growth

Working with me

Great user experience

Naming your icons

Conversations

All writing

The Enemy

The Hot Potato Process

Leave the Phone at Home

UX of Email Newsletters

Delight Comes Last

The Cost of Lies

Big Mood Machine

Simplicity is a war

The next iPhone

Working as a UX-lead

Leadership or management

Everyone should own a dog

Silence is gold

Cameras that understand

Humans, not users

Keeping AI Honest

Right to privacy

2018 in review

What’s my location?

The State of UX in 2019

Why scrap scrappy?

I’m taking a break

Organized for browsing

Stay humble, stay eager

The iPhone Franchise

Why Small Teams Win

Back to Work!

The Bullshit Web

Just keep at it

Let them eat cake

Netflix Culture

Skype

Unfoundered

Vanity Metrics

Tech is not Neutral

Productivity

Whose risk?

Why Small Teams Win

Phone Bored

Karim Rashid

Dieter Rams

Bleeding Out

Fake News is spam

Conversational Design

Dropbox

Bye bye Facebook

Cuba

The seat at the table

Givenchy

Love letters to trees

Pricing Philisophy

Specialize

Personas

Make me think

The Future of Retail

Hawaii Missile Alert

2017 review

Why Design Systems fail

What You Build

Checkout for Winners

Living a Testing Culture

What do you do?

Creative Class

How To Predict Your Future

Carpe Diem UX-designers

What´s Good Design?

Chasing Growth

Github

Medium

A Redesign

Design quotes

Enough

Why?

ARKit

Designing for Mobile

Is Less More?

Why Simple is Hard

Pricing It Perfectly

Failure, Reflect, Renew

Trusting Your Gut

Built to last

An Eye on the Future

Growth

Working with me

UX Design explained

Great user experience

Naming your icons

Conversations