Here’s How to Pick the Right Experimentation Methodology:
Many blog posts have been written on the topic of sequential testing, specifically from the standpoint of “why should you use sequential testing” and are filled with terms like “power” and “significance” and “null hypothesis” and I am here to tell you, dear reader, that this is not one of those blog posts. I am not a data scientist or a statistician and I can definitely not add anything to everything that has already been written. What I can tell you; however, is that while it is excellent in some cases, sequential testing is not always the best choice and Split lets you choose what works best for you. My goal is to give you real world use cases for each so you can make good decisions about how to run your experiments, without having to get a degree in data science (don’t worry though, a data scientist actually ran these numbers).
Before we get started, let’s cover the basics.
Split offers two statistical methods: Sequential Testing and Fixed Horizon
Sequential Testing is a statistical testing method where sample sizes are not fixed in advance. This means that at the beginning, you would have to see very extreme notable changes come through. With more data, we get a higher precision so that we can detect smaller effects. This allows you to make conclusions at a much earlier stage of the experimentation in comparison to the Fixed Horizon approach.
With Fixed Horizon, sample sizes and your experiment goals are defined in advance. In this method, you can draw conclusions only once your review period has completed. Since sample sizes are defined in advance, this is the perfect testing method when you’re looking to make small improvements to a relatively low-traffic site or page.
Let’s start with a situation where you really want to be using Sequential Testing. You’re a software engineer working on a relatively high traffic page on a feature that you’re hoping makes a big impact on one of your key metrics. You want to be able to check in on the rollout as it goes so you can stop it if anything bad happens, but you definitely don’t want to spend a bunch of time planning the exact number of users and conversions that you need to see in order to prove the project was a success. This is the dream scenario for Sequential Testing, you’re going to get your results faster, with less work, and less chance of false positives – since the results are always valid – than you would have with a fixed horizon approach.
Ok, but when should you definitely NOT use sequential testing? You are (or have access to) someone knowledgeable in experimentation or data science willing to make a specific test plan based on your traffic and goals, including how big of an impact you’re going to make and how many users you need to have in the test. You’re looking to make small incremental improvements to a relatively low traffic site or page. You’re willing to let the test run to conclusion to make sure the results are perfect, and you want to be able to detect even the smallest changes to your key metrics. In this scenario Sequential Testing would lead you to waiting longer for your results than with fixed horizon (or the results being entirely inconclusive), so fixed horizon is clearly the better choice.
There are also going to be times when you fall somewhere in between and it’s going to be a matter of preference. If you’re working on a low volume page but don’t have the knowledge to set up the experiment properly, maybe you still want to use sequential to ensure you don’t get any false positives. Or if you are working on a high volume site but really trying to detect even the smallest changes to your metric, perhaps you want to take the time to set up a fixed horizon test properly. Fixed horizon would let you see the impact faster in this scenario, but the Split sequential testing feature would still detect it, just with a little more traffic, which means more time.
The key points to remember are:
Generally speaking, sequential testing is better for:
- High volume of data, perhaps 500 or more uniques per version
- Quickly detecting major degradations
- No statistical pre-work
- Want to be able to look at results often
- Want a lower chance of false positives
- Not as interested in detecting small changes to the metric
And generally fixed horizon is better for:
- Low data volume, under 500 uniques per version
- Willing to do the work in advance
- Don’t need to check in on your results before they are ready
- Want to be able to detect very small changes to your metrics
- Aren’t terribly impacted by potential false positives
In Conclusion
Which path you should take for running your experiments is going to depend on many factors including the number of users in the test, how quickly you need your results, how big of an impact you believe this change will have, and whether or not you want to be able to “check in” on the test while it’s running. For many experiments Split’s new Sequential Testing functionality will be just what you need, but as we just discussed, many tests should continue being executed as fixed horizon. We hope that this gives you the power and flexibility to get to better answers faster than ever. Keep an eye on Split in the coming weeks as we have more to bring you on this topic soon.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Switch It On With Split
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Switch on a free account today, schedule a demo, or contact us for further questions.