When Apple introduced iOS7 last year people were wowed by its stark design and, of course, the new features. Unfortunately for some users, it was also making them sick. Specifically, some users reported headaches and nausea brought on by using the zoom and slide functionality and the floating icons above animating wallpaper.
TidePool mobile app developer Jenni Leder described it to The Guardian this way. "It's not apps that affect me, but accessing them. Tap a folder and the view zooms in. Tap an app and it's like flying through the icon and landing in that app's micro world -- and I'm getting dizzy on the journey there." This wasn't a case of people grousing. As The Guardian noted, 3D effects can evoke these symptoms for people with vestibular disorders.
Apple typically plays its cards close, but we can assume it tested iOS7 numerous times and in several different scenarios. So this example is particularly illustrative: If Apple can't run a perfect test of a new app how can your average everyday company?
Doing it right
The literature on this subject -- which is particularly rich and in-depth regarding multivariate and A/B testing for email and Web content --is discouragingly rife with all the ways such a test can go wrong. Perhaps the sample size was too small or there was an internal event that changed visitor behavior. Other variables to watch out for as you run these tests over and over, experts say, are slight changes in technical environments or measurement tools or even incoming traffic or traffic mix.
But the rewards of successful multivariate testing are well worth it. And it isn't rocket science either, says John McDougall, author of the Web Marketing On All Cylinders and president of McDougall interactive.
He told CITEworld about tinkering with the landing pages for a lead generation campaign for its IT support client. The landing pages were built using Unbounce, a hosted platform for landing page creation and testing. When they started, the cost per lead (CPL) was around $4.
McDougall began A/B testing the landing pages first by experimenting with headlines.
"Amazingly, simple changes to our headlines where we were more specific and to the point as to what our services would do for the consumer, led to a substantial improvement in conversion rates and our CPL."
The company also A/B tested the pay-per-click copy, with similar results, McDougall said. "For our PPC ads, we found simple changes to our copy, again being a little aggressive in the bottom line benefit statements of our service, helped to boost conversions at the very top of the funnel." Taken together, he said, these efforts shaved about $2 off of the CPL on a per lead basis.
McDougall also tested various "calls to action," to borrow the industry parlance, on web forms. Testing those elements, however, affirmed that the right wording was one the web form buttons, he said. But it's notable that those phrases were tested at all -- it's a surprisingly common mistake to overlook such minute details.
Applying the "test everything" rule to mobile
Now back to Apple. This same concept, which can be boiled down to "test everything," applies to mobile apps as well. Fortunately new products coming online are making this process easier.
Appboy's Marketing Automation for Apps platform, for example, supports testing for six variants across numerous channels including in-app messaging and push notifications.
Another example is SiteSpect, which recently released an SDK to perform A/B testing and behavioral targeting for native iOS apps.
Publishers Clearing House (PCH) is releasing a mobile-ad optimization service that lets users test advertising creative pieces, based on gender with attributes such as device, source and age to follow, according to Mediapost.
To free up room in the IT budget for these tools, some corporate buy-in will probably be necessary. McDougall advises having empirical evidence and projections at the ready. "If you are a marketing manager and trying to "sell" the idea of A/B testing landing pages, you might show the potential improvement in advertising ROI as a result of boosting lead conversion rates by another 40%," he says.
But before you go down that path, it is equally as important to have the basics down, he adds. "A/B testing will only bring you so far if you are missing key elements. Is your contact information easily available? Are testimonials easily found? Does your site load in a reasonable time frame? Is there a phone number and address?"
He adds, "You can A/B test headlines and colors until you're blue in the face, but if something key is missing or makes consumers feel they cannot trust your company, they will leave."
Or possibly throw up.