Television New Zealand (TVNZ) is New Zealand’s state-owned, commercially funded broadcaster. With television channels TVNZ 1, 2, and DUKE, as well as online streaming platform TVNZ OnDemand, they reach two million New Zealanders daily.
“One of our first experiments was a great learning experience. It prevented us from introducing something that we thought was good, but actually would have likely given us a negative impact. We’re now doing another iteration to get it right.”
Nathan Wichmann, Product Manager, TVNZ OnDemand
The Business Case To Accelerate Product Innovation
TVNZ OnDemand started with an in-house feature flagging system that required the DevOps team, rather than individual engineers or product managers, to turn features on and off. In addition to the feature flag system, TVNZ OnDemand had a manual experimentation process. An analyst would develop a list of users that they’d test a feature with and manually define the control and variation groups. This process was slow and it didn’t allow for a high cadence of experiments, random sampling, or high statistical rigor.
TVNZ decided to search for a solution that would allow them to speed up the ability to experiment and take out a lot of the manual work associated with it. The team wanted to scale from doing one experiment every month to several running experiments continuously so they could get customer feedback quickly and iterate.
TVNZ chose Split to build a solid feature flag foundation, extend experimentation to server- and client-side A/B tests, and to minimize long term maintenance and development costs. According to Mel De Soysa, TVNZ’s Business Analyst during the Split implementation, “Split allows us to get smarter around releasing software, by allowing us to toggle features on and off with the press of a button.” With Split, the TVNZ team can now quickly kill a feature that is having negative customer impact. “We can solve the problem and then figure it out later, rather than scramble and try to fix something on the fly with everyone involved.”
Mel also notes that Split’s experimentation capabilities “allow us to measure the impacts of these features so that we can match them up to our key engagement metrics, which lowers the risk of releasing the wrong thing or pushing out the wrong feature. At the same time, it helps us make informed decisions about the next feature or the next thing that we should look at for the product.”
“We use feature flags and run experiments a lot more because it’s easier now with Split. We can target a random sample size. We can increment the size of that sample really easily, and we can toggle it off immediately if there’s an issue.”
Mel De Soysa, Business Analyst
Speeding Up Experimentation and Insights
TVNZ OnDemand has been able to ramp experimentation quickly using Split. Most of the manual steps are gone and speed and quality have dramatically increased. Nathan Wichmann, Product Manager at TVNZ OnDemand describes that Split automatically assigns users into experiment treatment groups, ensures proper randomization, and handles the statistical calculations.
Once the experiment is run, the product and engineering teams now have a real-time view into how the experiment is performing. Nathan mentions that the team “can just log into Split and the metrics are recalculated every hour to every day. We can always get that information out at any time.” Instead of having to wait days or weeks for an analyst to analyze the results and come back with some insights, “with Split you know the answer straight away.” The TVNZ team can now decide to roll out or roll back a feature as soon as an experiment finishes.
“You move a lot faster. We’ve gone from running one experiment every few months to being able to run so many experiments that we have to think carefully about sequencing and prioritization.”
Nathan Wichmann, Product Manager
Split also helps TVNZ to do precise targeting of features and experiments. Early in their implementation with Split, the TVNZ team was working to personalize the homepage of the service based on certain viewer segments. “Split enables us to segment users dynamically, not just based on attributes at a static point in time,” Mel recalled. “Every time we wanted to change the variation or increase the sample size, we could just do that straight away.”
“Everyone has been so impressed with Split and how quickly we’ve been able to experiment and get our results. There’s a lot of excitement around using it to test our new features. Everyone that works directly with the product sees it as a great success.”
Nathan Wichmann, Product Manager
Connecting to TVNZ’s Data Pipelines
As TVNZ have ramped up experimentation, they’ve ensured that it is built on strong statistical practices and precise measurement. They identify metrics used in experimentation that align well core business metrics, ensuring that they are properly sensitive to individual product changes.
“The way that Split displays experiment results makes them really easy to understand and share. Our business teams log in and all the statistics they care about are there.”
Mel De Soysa, Business Analyst
TVNZ ingests event data from multiple endpoints and systems via Split’s Segment integration. The team also uses Segment to export data into their own data warehouse and into Google Analytics for post-processing. This has saved data engineering time and analyst time to run reports.
“It was super easy and smooth to use Split’s Segment integration. Within a week we had the integration running and events flowing.”
Mel De Soysa, Business Analyst
Running Experiments on New (and Existing) Features
TVNZ is building a culture of experimentation. The product team identifies the hypotheses behind a feature and looks to use A/B testing to help confirm these in practice. In cases where there is high confidence in a hypothesis or where “this is just clearly the right thing to do,” an A/B test may not be necessary. But with many new features, the team views the situation as “this is a good idea, but we don’t know for sure. Let’s design an A/B test.”
The OnDemand team has had several experiments where they were very confident in a good result but where Split revealed a neutral or negative result. In one example, they introduced a new piece of show metadata, commonly available within competitor apps, that they believed would add value for their viewers and increase engagement. Interestingly, the experiment revealed that this actually reduced some key metrics, forcing the team to rethink their assumptions, and iterate further to find a better solution.
In another example, TVNZ experimented with removing existing show metadata that they suspected cluttered the user interface without adding significant value. Being able to experiment and measure gave the team the confidence to do this, safe in the knowledge they could roll back if required. In this case their assumptions were validated, with no negative impacts observed, allowing the team to simplify the product without compromising the viewer experience.
Scaling Innovation As You Scale Your Product
Having an experimentation platform rebalances the relationship between having more features, more things in your product, more to support and the speed at which you can move. Nathan noted that “there are years-old features that people questioned the value of but feared removing. Having an experimentation platform allows us to revisit features and get broad business support to keep or remove them.” Split made it possible for TVNZ to take larger risks trimming feature-debt, providing the confidence knowing whether a feature has a positive or negative impact.
“We’ve been able to measure product impact and show that the ideas that we’ve worked on have produced tangible business results. Split has been really useful for us.”
Nathan Wichmann, Product Manager
If you’d like to learn more about Split and how we can help scale innovation in your organization, we’d love to chat. Or, if you’re ready to dig in and build your first feature flag, sign up for our free tier and get busy!
Switch It On With Split
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Schedule a demo to learn more.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.