Living a Testing Culture

November 10, 2017

Great notes from Luke W from Max van der Heijden's lessons learnt from the A/B testing culture at Booking.com.

Bullet points that stuck out to me:

  • Booking.com books 1.5 million room nights per day. That's a lot of opportunity to learn and optimize. More than 40% of all Booking.com sales happen on mobile.
  • Booking.com A/B tests everything. If something cannot be A/B tested, Booking.com won't do it. There's more than 1,000 A/B tests running at any time.
  • Teams are made for testing a hypothesis. They're assembled based on what resources they need to vet a hypothesis. They are autonomous, small, multi-disciplinary (designers, developers, product owners, copywriters, etc.) and have 100% access to as much data as possible.
  • Booking.com's experiment tool allows everyone to see all current experiments to avoid overlaps and conflicts between testing.
  • Failure is OK. Most of your learnings come from tests that don't work. 9/10 tests at Booking.com fail.
  • Everyone needs to to live the testing culture. Ideas can come from anywhere and go all of them go through the same testing process. Everyone is part of this.
  • Data only tells you what is happening. User research tells you why. These two efforts work together to define what experiments to run next and then test them.
  • What worked before might not work now. Always challenge your assumptions.
  • You can innovate through small steps.
  • Delighting customers doesn't necessarily mean more profit. Short term gains may not align with long term value. You'll need to find a balance.

It's evident that in order to make a testing culture work it needs to be implemented at all levels (designers, developers, product owners, copywriters) and everyone needs to learn from each other in order to not make the same mistakes over and over. One of the reasons for their success is to allow mistakes (90%) but to learn from them and share insights within the entire organisation.

Not everything needs to be big steps. Refining the copy of a button can make as much different as redesigning the entire page.

It’s honest, authentic, and accessible.

I love sharing my experiences working in design and what’ve I’ve learned along the way. Join a community of thousands of designers, developers, and product professionals by signing up to my newsletter!

Not quite ready to sign up? I totally understand! Why not start by reading some of my past issues?

Great! Just “one more thing”...

You need to confirm your email to confirm your subscription.

  1. March, 2024

  2. February, 2024

  3. January, 2024

  4. December, 2023

  5. November, 2023

  6. October, 2023

  7. September, 2023

  8. August, 2023

  9. July, 2023

  10. June, 2023

  11. May, 2023

  12. April, 2023

  13. March, 2023

  14. January, 2023

  15. December, 2022

  16. November, 2022

  17. September, 2022

  18. August, 2022

  19. July, 2022

  20. June, 2022

  21. May, 2022

  22. April, 2022

  23. January, 2022

  24. December, 2021

  25. November, 2021

  26. October, 2021

  27. September, 2021

  28. August, 2021

  29. July, 2021

  30. June, 2021

  31. May, 2021

  32. April, 2021

  33. March, 2021

  34. January, 2021

  35. November, 2020

  36. October, 2020

  37. September, 2020

  38. August, 2020

  39. June, 2020

  40. May, 2020

  41. April, 2020

  42. March, 2020

  43. February, 2020

  44. January, 2020

  45. December, 2019

  46. November, 2019

  47. October, 2019

  48. September, 2019

  49. August, 2019

  50. July, 2019

  51. June, 2019

  52. May, 2019

  53. April, 2019

  54. March, 2019

  55. February, 2019

  56. January, 2019

  57. December, 2018

  58. November, 2018

  59. October, 2018

  60. September, 2018

  61. August, 2018

  62. July, 2018

  63. June, 2018

  64. May, 2018

  65. April, 2018

  66. March, 2018

  67. February, 2018

  68. January, 2018

  69. December, 2017

  70. November, 2017

  71. October, 2017

  72. September, 2017

  73. August, 2017

  74. July, 2017

  75. June, 2017

  76. May, 2017

  77. April, 2017

  78. March, 2017

  79. February, 2017

  80. January, 2017

  81. December, 2016

  82. November, 2016

  83. October, 2016

  84. September, 2016

  85. August, 2016

  86. July, 2016

  87. June, 2016

  88. May, 2016

  89. April, 2016

  90. March, 2016

  91. February, 2016

  92. January, 2016

  93. December, 2015

  94. November, 2015

  95. October, 2015

  96. September, 2015

  97. August, 2015

  98. July, 2015

  99. June, 2015

  100. May, 2015

  101. April, 2015

  102. November, 2014

  103. April, 2013