In May 2014 development of Barnplay.se was winding down but two of my colleagues, Simon and Emil, was haunted by a feature they did not feel was implemented optimally.
All shows in Barnplay have episodes that kids can choose from. However, the shows also have a text about the show and some shows have other “functions”:
- sort the episodes differently
- a link to the shows web site on barnkanalen.se
- mark the show as a favorite or remove from favorites (only in our apps)
These “functions” where presented with a round icon and text and Simon and Emil thought this could become better. A smaller representation for the functions would also make the functions fit better in phones held in portrait position. Their suggestion was a “button” look without a text label.
I however had already via usability tests proved that icon+text works. So I did not want to change to a “button” look without first proving that it was better. I decided to do an A/B test in our tool Optimizely.
While Emil created graphics for the “buttons” and Simon made them possible to display in Barnplay I set up the experiment in Optimizely. The variations in the experiment where “icon+text” versus “button”. We want the functions to be used in general but ultimately want them to generate video starts. For goals I therefore choose to track:
- taps on any of the functions
- taps on each of the functions
- a two step activity where the visitor has first clicked “Sort” and then started an episode
We ran the experiment for 2 days which resulted in 142 000 visits and lot’s of taps:
|Tap on any function|
|Icon+Text||70829||2536||3.58% (+/- 0.14)||—|
|Button||71117||9155||12.87% (+/- 0.25)||+259.5%|
|Tap to read about show|
|Icon+Text||70829||1992||2.81% (+/- 0.12)||—|
|Button||71117||5584||7.85% (+/- 0.20)||+179.2%|
|Tap on link to show site|
|Icon+Text||70829||393||0.55% (+/- 0.05)||—|
|Button||71117||474||0.67% (+/- 0.06)||+20.1%|
|Tap to make/remove favorite|
|Icon+Text||70829||5914||8.35% (+/- 0.20)||—|
|Button||71117||4119||5.79% (+/- 0.17)||-30.6%|
|Tap to sort episodes|
|Icon+Text||70829||1815||2.56% (+/- 0.12)||—|
|Button||71117||2769||3.89% (+/- 0.14)||+51.9%|
|Tap to sort, then start episode|
|Icon+Text||70829||800||1.13% (+/- 0.08)||—|
|Button||71117||1422||2.00% (+/- 0.10)||+77.0%|
All shows have the “Read about the show” function which is why that function has so many more taps then other functions. The stunning difference between the text+icon and button variants is a strong indicator that button works better. Link taps where quite few so the result is not so interesting. That the visitors used the favorite function less is troubling. The sorting and sort+start goals however look very promising.
Despite the decrease in activity on the favorite function, buttons was chosen as the winning variant in this test. Simon implemented them fully and they are now in production.
So, I had to change my mind and I love when that happens! :)