A/B testing in general is the process of testing two variations of a resource by showing different versions to different users, then comparing the test results (aka, the differences in user behavior between the two groups) for statistical significance. The process is essentially the experimental method as applied to software development.
There are two things that people mean when they say “mobile A/B testing”: app store A/B testing and mobile app A/B testing. Today, we’ll focus primarily on the latter. A/B testing for mobile apps is about as similar to standard A/B testing as mobile app development is to standard software development. There are some key differences (for example, mobile apps have unique features like push notifications that developers can and should use), but the overall methodology is the same.
The Mobile A/B Testing Process
Mobile A/B testing, as with any experiment, begins with data gathering. What are your current metrics and what are some key areas of your app that could improve along those metrics? For example, if you have in-app purchase, how many of your app users are making it? If your app has complex functionality, are too many people dropping off in the user onboarding phase? After data gathering comes hypothesis formulation: what could you change to fix these user engagement problems? What re-engagement strategies could you use? After deciding on what hypothesis you’d like to test, you build the new variant, split your total user base in two, and serve one variant of the feature to each group.
Many standard A/B testing tools can be used for mobile applications, but ideally, developers should buy or build tailor-made mobile A/B testing tools. This will let them take advantage of the unique aspects of the mobile experience which are some of the reasons they built a mobile app in the first place. It will also let them account for unique drawbacks of mobile development: somebody in the city will have lower network latency than someone in the country, but the app should work as well for both of them. Further, a lot of mobile A/B testing tools have built-in visual editors which make it easier for developers to design something that actual mobile users will want to use. After all, how many times have we all designed new features for our apps which looked really cool on a simplistic desktop simulation, but that ended up looking awful on a real smartphone screen?
Though the process is similar for A/B testing mobile apps as it is for any other software, understanding the unique aspects of the mobile development process and keeping those in mind during your testing process is paramount to designing and maintaining a great app.
