An A/B Test that made me change my mind

Part 4 of 9 in the series Deliverables.

In May 2014 development of was winding down but two of my colleagues, Simon and Emil, was haunted by a feature they did not feel was implemented optimally.

All shows in Barnplay have episodes that kids can choose from. However, the shows also have a text about the show and some shows have other “functions”:

  • sort the episodes differently
  • a link to the shows web site on
  • mark the show as a favorite or remove from favorites (only in our apps)

These “functions” where presented with a round icon and text and Simon and Emil thought this could become better. A smaller representation for the functions would also make the functions fit better in phones held in portrait position. Their suggestion was a “button” look without a text label.

I however had already via usability tests proved that icon+text works. So I did not want to change to a “button” look without first proving that it was better. I decided to do an A/B test in our tool Optimizely.

While Emil created graphics for the “buttons” and Simon made them possible to display in Barnplay I set up the experiment in Optimizely. The variations in the experiment where “icon+text” versus “button”. We want the functions to be used in general but ultimately want them to generate video starts. For goals I therefore choose to track:

  • taps on any of the functions
  • taps on each of the functions
  • a two step activity where the visitor has first clicked “Sort” and then started an episode

We ran the experiment for 2 days which resulted in 142 000 visits and lot’s of taps:

Tap on any function
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 2536 3.58% (+/- 0.14)
Button 71117 9155 12.87% (+/- 0.25) +259.5%
Tap to read about show
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 1992 2.81% (+/- 0.12)
Button 71117 5584 7.85% (+/- 0.20) +179.2%
Tap on link to show site
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 393 0.55% (+/- 0.05)
Button 71117 474 0.67% (+/- 0.06) +20.1%
Tap to make/remove favorite
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 5914 8.35% (+/- 0.20)
Button 71117 4119 5.79% (+/- 0.17) -30.6%
Tap to sort episodes
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 1815 2.56% (+/- 0.12)
Button 71117 2769 3.89% (+/- 0.14) +51.9%
Tap to sort, then start episode
Variation Visits Taps Conversion Rate Improvement
Icon+Text 70829 800 1.13% (+/- 0.08)
Button 71117 1422 2.00% (+/- 0.10) +77.0%

All shows have the “Read about the show” function which is why that function has so many more taps then other functions. The stunning difference between the text+icon and button variants is a strong indicator that button works better. Link taps where quite few so the result is not so interesting. That the visitors used the favorite function less is troubling. The sorting and sort+start goals however look very promising.

Despite the decrease in activity on the favorite function, buttons was chosen as the winning variant in this test. Simon implemented them fully and they are now in production.

So, I had to change my mind and I love when that happens! :)

Barnplay Functions: A/B test result

Leave a Reply

Your email address will not be published. Required fields are marked *

From ”Let us build” to ”Let us understand (and maybe build)”

Sometimes a project is started based on a feeling that a designer, engineer or manager has. However, what we choose to build should be "directed" by our users. Is there a way of getting user driven in a project that started based on internal feelings? I think Krste has found a way!

We are eating the software creation process

I think a transformation is happening in digital product creation: We ux/product designers are eating the software creation process. It's a good development and your team should take advantage of it. Marc said...

Pac-Man by Namco.

Published in process.