I recently (ok, many weeks ago, just lazy to write my thoughts) participated in small a/b testing seminar, which was organized by Reaktor, hosted by @SimoAhava and presented by @OptimiseOrDie. It was a nice powerpoint show by Craig and audience surely realized that he is very passionate about a/b testing and he shared very good tips, tricks and links.

Someone asked what kind of advise Craig would give if you are starting a/b testing in a big organization and not really getting anyone else excited about testing and/or if it’s considered risky to start this strange testing process. To be honest, I don’t remember what was Graig’s answer, but I thought I could also have some tips to share because I started a/b testing process from scratch in my organization little over year ago. I’m still not 100% happy about our process and what we have accomplished, but surely have learned a lot and here are my random tips for different situations.

Get your confidence level high with the tool

I prefer Optimizely, but you can choose whatever is right for your needs and many times this is just a question what kind of user interface you like etc.

When you have decided the tool, start testing with your own blog or with some dummy site you have access. Get the confidence to use the tool before starting real testing with company sites. Not a technical person? Nowadays, the testing tools are supereasy to use, at least for easy testing purposes, but you can always get some consultancy help to get started or to do the implementation process.

When the confidence level is high enough you can start testing with real sites and hypothesis. Don’t be a shame if you start with some lame testing like I did, this is your first test and to me it sounds success if the first test is technically done ok and nothing breaks. You’ll surely notice quickly if you are always testing stupid stuff, but first test is ok to be boring.

Find partner in crime and make him/her the star of the show

What site? Where to really start? Try to find product owner who is at least little bit interested about a/b testing, because it is important to get internal support before you can really test something. As soon as you get some success with testing make sure you give the credit also to the product owner and try to make he or she as the star of a/b testing process -> Make an internal success story -> and soon there is a line of product owners waiting at your door to get this magical a/b testing to get started on their pages/products/websites…

Make testing fun, there is no better way to get “internal publicity” to a/b testing than making it fun and exciting in positive way. For example, I once said that my hypothesis is 100% accurate and if my test variation is not going to win the test then I will bake chocolate cake for my colleagues… guess who had to bake the cake, well, it was good anyway.

Highlight & follow the results

Yes, you need statistical significance to get real results etc, but sometimes the amount of conversions aren’t very high in numbers. However, if you put some perspective to the numbers and calculate the conversion improvement based on longer period of time and that way e.g. 1000 conversion improvement in month could be 12,000 more in a year. You have to be careful what you are “promising” and of course take many other things to considerations when making these kind of predictions, but you get the idea how to put some shining on the numbers.

Keep track on your tests. Make an excel and there you put all the information about your tests; what test, hypothesis, starting day, end day, results with comments, and was something new hard coded to the site based on the test etc. This way you can easily show what you have accomplished and hopefully you get to see and compare your testing success for the earlier years.

Advanced Tips

This has got nothing to do with the question “how to get started”, but will share anyway some random advanced tips

Don’t just test, test for right purposes -> Use hypothesis

Don’t make variations that you are not 100% happy about, meaning if the variation wins, don’t say that it can’t be hard coded to the site for reason X. Why the hell you even tested that variation. This usually means that you have been too busy to make the test and the variations are missing some graphical content etc why those look so bad or something.

Before going live, test the variations with most used browsers and devices.

AND THERE YOU GO, AS LONG AS YOU KNOW HOW TO BAKE CAKES YOU CAN START THE A/B TESTING! 😉