Sequential Testing is a statistical testing method where sample sizes are not fixed in advance. In Sequential Testing, boundaries are set, large at first, and smaller as more data comes in. This means that at the beginning, you would have to see very extreme notable changes come through.
With more data, we get a higher precision so that we can detect smaller effects. This allows you to make conclusions at a much earlier stage of the experimentation in comparison to the Fixed Horizon approach.
Why Implement Sequential Testing?
Sequential Testing is perfect for those teams that are looking to move faster without breaking anything in production. Results are always valid, which means you can peek at your data as many times as you’d like without waiting for your experiment review period to complete. This enables faster monitored rollouts, faster release decisions, and quicker iterations of experiments when big, notable changes are detected.
Sequential Testing vs. Fixed Horizons
Both Sequential Testing and Fixed Horizons address very different use cases. With Sequential Testing, results are always valid, which means you run a low risk of interpreting a false positive. This method is the most optimal when you’re releasing net new features and looking for big, notable changes within your data to determine whether it’s safe to move forward (or backward) with your releases.
With Fixed Horizons, you have to wait for your experiment review period to complete before peeking at your data to avoid interpreting any false positives. Fixed Horizons is great when you’re trying to make small optimizations within your product such as improving the page load time by 0.10%. You should use both approaches interchangeably, depending on your team’s needs. Learn more about setting up Sequential Testing here.
