The most listened to TED talk of all time is “How Great Leaders Inspire Action” by Simon Sinek. The point of Simon’s talk is that leaders focus on having a powerful “why” to guide the activities of their organizations, rather than trying to communicate lots of “what” and “how.”
Those of us who believe in product experimentation as a way to increase engineering impact and progressive delivery as a way to reduce the risk of moving fast would do well to heed that advice. You don’t have to go far to look for role models: start with Sonali Sheel of Walmart Labs.
Sonali works on the team that builds the Walmart Grocery mobile app. They started with a pilot in 2013 and are now in the process of rolling out to every Walmart in the nation. That puts their reach within 10 miles of 90% of the US population. With the Walmart Grocery app, customers shop on their phones and then just swing by the nearest Walmart to have the groceries loaded in the car.
Sonali recently spoke at a meetup the Split team put on with our industry friends in Santa Monica. Sonali briefly explained that Walmart built their own experimentation platform called “Expo” back in 2014 because commercial solutions available at the time didn’t fit their needs.
Rather than give us a deep-dive into the internals of Expo, Sonali focused on the two big “whys” behind her team’s use of it: “Test to learn” and “Test to launch.”
Test to Learn
The “Test to learn” pattern should look familiar to you if you’ve heard about A/B testing, conversion rate optimization, growth engineering or “growth hacking” approaches.
At Walmart Labs, “Test to learn” is about the iterative discovery of what works and what does not, or as Sonali put it in her presentation, to “Understand customer behavior and validate or invalidate a hypothesis.” Split’s Experimentation module focuses on this use case.
We won’t launch this feature until we can prove that it does what we want
Test to learn isn’t just a “research” tool, but rather a non-optional step in designing and shipping any new feature. Sonali’s team is only allowed to launch a new feature to the entire user population after they have proven that it delivers an experience that is better than the status quo. The goal here is to have a hypothesis about what will improve the desired customer behavior pattern, test it out to see whether it really does move the needle, and repeat the hypothesis/experiment loop until finding a variation on the feature that succeeds with enough real users.
Test to Launch
The “Test to Launch” pattern should look familiar to you if you’ve heard about gradual rollouts, canary releases and progressive delivery solutions like Split’s Feature Delivery Platform.
We will launch this feature unless it does something unexpected that we don’t want
Test to launch is about gradually rolling out a new feature to the entire population while keeping a close eye on “do no harm” metrics and stopping the rollout early if metrics degrade. As Sonali put it, Test to launch is first and foremost about mitigating risk.
Test to launch uses the same underlying capabilities of an experimentation platform, including the ability to manage selective exposure of a new feature and observe the system and user behavior differences between those who get a feature and those who do not. When used for launches, this pattern is like the monitoring module of the Split solution: it focuses on shorter time frames in order to accomplish a limiting of the blast radius of unintended consequences.
Engineering for Impact / Doing No Harm
Sonali’s team has an amazing opportunity to serve the world around them. With that opportunity comes a commitment to not just release features for the sake of proclaiming “done” but to ensure that features actually deliver impact and don’t do any harm to key business metrics. Whether you are building a business or widening an already formidable moat, you can use the same tactics to ensure your engineering efforts make a difference you can be proud of.
Learn More About Progressive Delivery and Experimentation
Liked this post? Make sure you check out these great resources to learn more and get inspired to experiment!
- Simon Sinek’s famous TED talk How Great Leaders Inspire Action
- Sonali and expert speakers from Tinder, Snap and Split explain how they use experimentation for competitive advantage in this video replay of Split’s Startup Coil event in Santa Monica last September.
- Adil Aijaz explains how To Feature Flag is to Experiment.
- The Split team shows that hope is not a strategy when it comes to feature flags and experimentation.
Looking for more great content on experimentation and feature delivery at scale? Bookmark the Split blog, check us out on YouTube, or follow us on Twitter @splitsoftware.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Switch It On With Split
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Switch on a free account today, schedule a demo, or contact us for further questions.