Living a Testing Culture

Great notes from Luke W from Max van der Heijden’s lessons learnt from the A/B testing culture at Booking.com.

Bullet points that stuck out to me:

  • Booking.com books 1.5 million room nights per day. That’s a lot of opportunity to learn and optimize. More than 40% of all Booking.com sales happen on mobile.
  • Booking.com A/B tests everything. If something cannot be A/B tested, Booking.com won’t do it. There’s more than 1,000 A/B tests running at any time.
  • Teams are made for testing a hypothesis. They’re assembled based on what resources they need to vet a hypothesis. They are autonomous, small, multi-disciplinary (designers, developers, product owners, copywriters, etc.) and have 100% access to as much data as possible.
  • Booking.com’s experiment tool allows everyone to see all current experiments to avoid overlaps and conflicts between testing.
  • Failure is OK. Most of your learnings come from tests that don’t work. 9/10 tests at Booking.com fail.
  • Everyone needs to to live the testing culture. Ideas can come from anywhere and go all of them go through the same testing process. Everyone is part of this.
  • Data only tells you what is happening. User research tells you why. These two efforts work together to define what experiments to run next and then test them.
  • What worked before might not work now. Always challenge your assumptions.
  • You can innovate through small steps.
  • Delighting customers doesn’t necessarily mean more profit. Short term gains may not align with long term value. You’ll need to find a balance.

It’s evident that in order to make a testing culture work it needs to be implemented at all levels (designers, developers, product owners, copywriters) and everyone needs to learn from each other in order to not make the same mistakes over and over. One of the reasons for their success is to allow mistakes (90%) but to learn from them and share insights within the entire organisation.

Not everything needs to be big steps. Refining the copy of a button can make as much different as redesigning the entire page.

Get more writing like this

Sign up and get new writing, just like this, every other two weeks. Unsubscribe any time (I'm not a dickhead).

Latest writing

Delight Comes Last

The Cost of Lies

Big Mood Machine

Simplicity is a war

The next iPhone

Working as a UX-lead

Leadership or management

Everyone should own a dog

Silence is gold

Cameras that understand

Humans, not users

Keeping AI Honest

Right to privacy

2018 in review

What’s my location?

The State of UX in 2019

Why scrap scrappy?

I’m taking a break

Organized for browsing

Stay humble, stay eager

The iPhone Franchise

Why Small Teams Win

Back to Work!

The Bullshit Web

Just keep at it

Let them eat cake

Netflix Culture

Skype

Unfoundered

Vanity Metrics

Tech is not Neutral

Productivity

Whose risk?

Why Small Teams Win

Phone Bored

Karim Rashid

Dieter Rams

Bleeding Out

Fake News is spam

Conversational Design

Dropbox

Bye bye Facebook

Cuba

The seat at the table

Givenchy

Love letters to trees

Pricing Philisophy

Specialize

Personas

Make me think

The Future of Retail

Hawaii Missile Alert

2017 review

Why Design Systems fail

What You Build

Checkout for Winners

Living a Testing Culture

What do you do?

Creative Class

How To Predict Your Future

Carpe Diem UX-designers

What´s Good Design?

Chasing Growth

Github

Medium

A Redesign

Design quotes

Enough

Why?

ARKit

Designing for Mobile

Is Less More?

Why Simple is Hard

Pricing It Perfectly

Failure, Reflect, Renew

Trusting Your Gut

Built to last

An Eye on the Future

Growth

Working with me

UX Design explained

Great user experience

Naming your icons

Conversations

Due to a late cancellation on a project, I have availability from August 19. Let's talk!

close