I attended a MeetUp in Ann Arbor, Michigan hosted by Human<>Element on Tuesday, June 3rd 2014. The topic being discussed was A/B testing How to Start Small and Win Big with A/B Testing. This MeetUp was by far the most informative and had the best goodies thus far.
Human<>Element is a web development company that has been in business for 10 years, specializing in making eCommerce websites using the eCommerce Magento platform.
Chalk Fly is an eCommerce website where you can buy office supplies and designate teachers to receive 5% of the total purchase to spend on school supplies.
Josh Frank who is the eCommerce lead at ChalkFly.com gave his expertise on A/B testing. Having never running a full blown A/B test myself, this was an interesting topic. I learned how powerful Optimizely is. Some notable topics Josh talked about that interested me were; quantitative vs. qualitative data, the 'Submit' button, the '300 Million Dollar Button' and common pitfalls in A/B testing.
Quantitative vs. Qualitative Data
Quanatitaive data like analytics tracking and event triggers are great ways to track customer behavior, although you also need qualitative data from your customers. Surveys make great qualitative data to get the experience coming from the source, look out for outliers though (overly angry / immature comments).
The 'Submit' Button
I never pay attention to the 'Submit' button anymore, I just click the button at the bottom of forms to proceed. Josh suggested we never use 'Submit', use anything else like 'Send Email!' or 'Start your free trial!'. This is also something I am going to start incorporating into the websites I create.
The 300 Million Dollar Button
Best Buy was the first eCommerce website to allow customers to checkout with first collecting their email, taking payment and then asking them to create a password for their account IF they want too. Best Buy saw revenues of 300 Million more in a year with this newly innovated approach to the shopping cart. This is why its now called 'The 300 Million Dollar Button'.
Common Pitfalls in A/B Testing
Short Tests - Recommend you run your A/B tests at least a full week. A good test would last 6 months.
Changing Elements - Changing things such as text and color during an experiment is a no-no.
Over Optimization - Don't go through a total redesign of your website just because you think its time. Run A/B tests to see if that type of redesign will work for your website.
Beware of Averages - Looking at averages of your sales will sometimes give you false data. For example you can have one corporate client buying products in the thousands of dollars and the rest could be regular people buying a few dollars worth of product.