Hippos are the Most Dangerous Animals on Earth
Read on to see What HiPPOs Have to do With UX Design
Photo Credit: P. Brundel

It’s hard for some people involved with web/app design and development to accept that they can’t rely on intuition, judgment or even qualitative testing methods to reliably predict the success of a new feature. When I worked at Microsoft, one site manager told us his team didn’t need to A/B test their site redesign because “Good design is in our DNA!”

My A/B testing colleagues and I have been tracking expert business, design and technical predictions for years and we have found that the experts, no matter how much experience they have or how highly paid they are, cannot reliably predict user and customer preferences. Despite this body of evidence, many people persist in acting as if such a priori opinions are reliable. This is The HiPPO Syndrome. HiPPO = Highest Paid Person’s Opinion: an acronym that was first popularized by Avinash Kaushik and by our Microsoft Experimentation Platform team. I designed HiPPO stress toys, posters and web pages that helped popularize the acronym.

If you hear a designer or site owner confidently declare that a new design doesn’t need A/B or multivariate testing because they created it very carefully using their very best secret sauce design practices and usability testing, run away as fast as you can. You are in the presence of HiPPOs, the most dangerous animals on earth.

Some designers and managers have attacked A/B testing because of a misconception that testing somehow replaces or diminishes design. Nothing could be further from the truth. Good design comes first. Testing can only verify that a design is valued by end users. The key is that it is the end users who are the final arbiters of design success, at least in sites and applications that live by traffic volume, engagement, loyalty and/or conversion.

When I worked on the Microsoft Experimentation Platform, we carefully tracked the results of A/B and multivariate tests over time and compared actual versus expected outcomes. We found that experienced UX designers and analysts predicted the correct outcomes about 1/3^rd of the time, the results were ambiguous 1/3^rd of the time and people were dead wrong 1/3^rd of the time. Colleagues who had previously worked at Amazon reported that the same approximate ratio was also true there. A quote from a Tnooz article on A/B testing of online Hotel content reinforces our point. A Posadas Hotels’ Sales Manager is quoted as saying:

“We learned that our opinion and the customer’s opinion about our best photo are not the same. We saw a 46% improvement in click-through compared to the image we originally were using. It definitely blew us away.”

If you don’t believe me and think you can predict design success significantly better than 1/3^rd of the time, I invite you to visit WhichTestWon.com. Each week, the site presents a completed A/B test and asks its visitors to guess whether they think version A or B won before showing the actual results. Have a look at some of the past tests and see how you do. (The site has put many past results behind a paywall, but there are still some you can browse for free. You just have to click through the link to each test to see if it is paywalled or not.) Of course, if after trying the site out, you find it useful, I encourage you to buy a subscription. In any case, you always subscribe to their free weekly newsletter.