I’m in service development and my main job is to do reports and develop different online projects, including mobile, with the help of web analytics. I have also taken the responsibility of A/B testing. With the help of our analytics partner we tested few possible A/B testing tools and at the end we choose Optimizely.

There are many other good tools, but there was just something special with Optimizely. Optimizely was for us super easy to use, price-quality ratio excellent and one of the biggest thing was easy Adobe SiteCatalyst integration. That’s why we started to pay for the tool after 30-day free trial period. I recommend to investigate more from Optimizely’s website, but I want to share my few tips after couple of tests with A/B testing tool called Optimizely.

Newbie meets customer service

No matter how simple any tool is, there surely is always some doubts how things works and newbie questions will rise from the dust and mist. Don’t you just hate when you contact to some customer service and their response is always “yes”, “no” or “here is the link”? Have to give credit to Optimizely’s customer service, they rock and take their job seriously. They really want to stand for the slogan “A/B testing you’ll actually use”. Even on the free trial period I got very detailed answers to my questions and few times they even take the time to login to our account and if needed they tested and double-checked everything.

Things to remember

• Remember to enable filtering after starting new project because IP filtering is per project.
• Remember to enable Adobe SiteCatalyst integration after every test because integration is per test. I guess this tip concerns every other analytics tools too.
• One thing that positively surprised me was the cross-browser test. One of the biggest thing for me when started to make tests was the question “are my variations going to work all the different browsers and devices that are available!?” and with help of cross-browser test I’m confident to publish my tests without the help of our scrum team.

Perfect tool for a/b testing, well, there’s this one thing

…that bothers me a lot. It’s the IP filtering. The tool is so easy to use, but then you see this you say in your mind “woot!?”.

I’m sure that everyone has to use this filtering, at least, to filter internal ip addresses from the tests. That’s why you would think that this part should be super easy to do and quickly! Yes, words like “easy” and “quick” are not possible in sentence where is also “regular expression”. Don’t get me wrong, I love regular expression in different situations but not for this. This could be one option, but why isn’t there any other options like “import .cvs file with ip addresses” or simple fields to copy and paste ip addresses.

I can understand that there is limitation for characters, but with regular expressions this limit is reached too quickly. Big companies have many ip addresses that’s need to be filtered (many internal and different agencies to filter) and with that limitation we are not able to filter all that we need to. Thanks to SiteCatalyst integration we are able to analyze our test results with better ip filtering, so at the end this was not issue why we couldn’t use the tool. Yes, I gave feedback about this issue and Optimizely’s development team promised to make improvements and I’m very happy to hear that. Optimizely, there’s no pressure but we are waiting. 😉

Getting party a/b testing started

It’s totally free for 30 days so why not to test and the smallest account is cheap as… well, very cheap. A/B testing = always be testing!

If you are new to conversion rate optimization you can start by testing pretty much everything, but remember that in the long run you should only test things that matters. I have also done mistakes by testing issue X and after the test I said to myself “that was fun test, but I’m surely not going to change anything on our site based on the results”.

You should make hypothesis for every test, measure actionable metrics and wait for statistical significance, say what? Well, you can start learning from here http://en.wikipedia.org/wiki/Conversion_optimization and with Google you can find all the information needed to becoming a/b testing guru. Optimizely also shares good guides and templates to start keep a record of your tests.