The A/B Test: Inside the Technology That’s Changing the Rules of Businessv

Source: http://www.wired.com/business/2012/04/ff_abtesting/

A/B tests have been done in engineering companies for a while and are becoming popular inside companies such as Google. The goal of A/B test is to “ understand the impact of some change in the website UI, underlying algorithm, or marketing campaign for example, on users or customers”.This article gives examples how Obama campaign team used A/B test to better customize their website in order to increase subscription and donations.

 The new-media team already knew that their greatest challenge was turning the site’s visitors into subscribers—scoring an email address so that a drumbeat of campaign emails might eventually convert them into donors. Their visit would start with a splash page—a luminous turquoise photo of Obama and a bright red “Sign Up” button. But too few people clicked the button. Under Siroker’s tutelage, the team approached the problem with a new precision. They broke the page into its component parts and prepared a handful of alternatives for each. For the button, an A/B test of three new word choices—”Learn More,” “Join Us Now,” and “Sign Up Now”—revealed that “Learn More” garnered 18.6 percent more signups per visitor than the default of “Sign Up.” Similarly, a black-and-white photo of the Obama family outperformed the default turquoise image by 13.1 percent. Using both the family image and “Learn More,” signups increased by a thundering 40 percent.

The result is surprising and illuminating. This kind of experiments give these digital company a chance to utilize their marketing strategy in a very low cost. It also arise the attention about how far the reality might be from our assumption. Some examples are given in this article.

Most shocking of all to Obama’s team was just how poorly their instincts served them during the test. Almost unanimously, staffers expected that a video of Obama speaking at a rally would handily outperform any still photo. But in fact the video fared 30.3 percent worse than even the turquoise image. Had the team listened to instinct—if it had kept “Sign Up” as the button text and swapped out the photo for the video—the sign-up rate would have slipped to 70 percent of the baseline.

Although this kind of experiments are very hard to realize for offline marketing, it might be a trend in utilize digital marketing strategies. As concluded in the article:

What this means goes way beyond just a nimbler approach to site design. By subjecting all these decisions to the rule of data, A/B tends to shift the whole operating philosophy—even the power structure—of companies that adopt it. A/B is revolutionizing the way that firms develop websites and, in the process, rewriting some of the fundamental rules of business.