Ordinary software development is widespread with guesswork. Some amount of data, plus some amount of the product manager’s hunches, drives the decisions of what new features the development team will create. Without some fairly major shifts in how development is done, this is almost how it has to be: you have to guess how your users will react to a feature, because you can’t just show it to them and gauge their reaction.
With feature experimentation, this dynamic shifts significantly. The product manager is able to make data-driven decisions based on the actual performance of the features in production, because the new features are actually deployed to (a small percentage of) the real user base. Now, instead of guessing user reactions, the product team can deploy the feature in production to a small number of users, using customer experience surveys and KPI tracking to determine the feature’s effectiveness, or even A/B testing or multivariate testing different versions of features to find the best one.
The Benefits of Feature Experimentation
There are two central benefits of feature experimentation: increased understanding of the user experience, and increased ability on the part of the development team to improve it.
No matter how well you know your user base, you won’t be able to predict their reactions perfectly. Instead of trying to do this impossible thing, many teams have switched to testing new features in production. When you can gather real-time data on how a new feature has impacted your KPIs — be they conversion rate, page load time, or API response time — you can choose which features to release to all your users based on a high probability that those features will produce a positive business impact.
Implementing Feature Experimentation
If experimenting with features is so great, how do you go about doing it? The most common way to do any type of product experimentation is to use feature flags, which allow you precise, granular targeting on who sees what features. Using a feature flag-based experimentation platform (like Split), anyone — the marketing team, the product team, the development team, the product management, or anyone else — can toggle features on and off.
Feature flags have a host of other use cases as well: they enable continuous delivery, promote DevOps, and allow you to monitor features even after you deploy them to your whole user base.
