Data Points
- Increased revenue by launching experiments every two weeks that drive business growth such as increasing conversion rates and entitlement upgrades.
- Freed developer and product manager time by eliminating the manual creation and tracking of feature flags.
- Improved release velocity by helping all development teams improve their code branching strategy to support continuous delivery.
The Situation
Before deploying Split, Envoy’s process for managing new functionality rollouts was slow and cumbersome. This caused both the product and development teams a great deal of frustration, as they had limited flexibility and control over the process. The team realized they needed to implement a phased rollout process to reduce the risk of production failure, and started testing out feature flagging with an in-house solution based on a modified Ruby gem library.
However, product managers (PMs) still had to work directly with a developer on every feature rollout, which caused significant coordination overhead. PMs had to manually compile a list of customer IDs for feature exposure. Just setting up entitlements for beta testing was complicated, let alone pushing out into a phased production rollout. Expanding the customer segment for a phased rollout required another manual compilation of customer IDs, and so on until they finally reached 100% of their customer base.
The team also had limited visibility into what treatments were being served and to whom. Envoy is very design-minded, and keeping track of which treatment a customer receives is critical during a rollout, especially for those customers who may be considered high risk or high touch. Additionally, not knowing what experience a customer was receiving made customer support more troublesome than necessary.
It’s often common for development teams to start with this type of internally developed feature flagging system to prove out the concept across their teams. Envoy was experiencing rapid business growth and quickly realized that their in-house solution was not going to scale with the needs of their business. Split’s turnkey solution was the perfect next step.
“Before Split, our feature releases were cumbersome and time-consuming, requiring our product managers to make special requests to developers every time we wanted to launch new functionality.”
Eric Crane, Product Manager
The Results
Using Split’s feature flagging capability to manage all their new feature rollouts has become part of Envoy’s standard release process today, significantly mitigating the potential risk of production failure. Envoy wraps every feature in a split prior to release, enabling them to conduct functional and performance tests in production without impacting their entire user base.
“Every team – from product and engineering to design and support – leverages Split to confidently create a great experience for our users.”
Eric Crane, Product Manager
Envoy set up Split feature flags across all their code repositories, including back-end and front-end for both desktop and mobile applications. In the past, keeping their deployment systems in sync had been problematic. Today, Split helps them manage the timing of their rollouts and gives them confidence that new functionality across all their systems and user devices are simultaneously rolled out. With tens of thousands of daily users, maintaining consistent functionality is critical!
Split has also made Envoy’s deployment process significantly easier now that product managers don’t have to submit a product request or wait for a developer to become available to help out – they can just use Split to select and go! For most features, they will start a feature rollout at 25% exposure. Once this has passed test criteria, they will ramp to 50%, and then 100%. For changes considered low risk, such as an update to the global delivery log, they will ramp even faster and quickly clean up the split.
Features deemed ‘risky’ by the development team will be assigned a more cautious rollout schedule. For example, some features may involve a giant branch of code that directly impacts infrastructure. Envoy sets up test environments using Split, giving teams access to new functionality prior to any production deployment. Once the team is ready to push into production, the new code branch becomes one split that lives for a long time over a cautious rollout schedule.
Performance Testing
A major concern for the team is how new functionality will impact performance. Some Envoy customers have very large contact lists that need to be uploaded. Loading these large files can skew performance results during the initial evaluation. With Split, Envoy product managers are able to easily define the target segments, giving them better control when evaluating the impact of the change on performance and infrastructure. By leveraging segments, they are able to avoid customers that fall into outlier categories during the early rollout phases.
Beta Programs
Product managers at Envoy often run a beta program as a limited release to specific customers. Having an on/off switch speeds rollbacks for resolving issues that occur during testing. Anything that causes a problem for their customers or increases support calls will be immediately turned off. Product managers and test teams are confident that with just one click of a button they can revert back to a previously successful application behavior.
Branching Strategy
Using Split has helped developers at Envoy improve their branching strategy to support continuous delivery of new functionality at a sustainable pace. In-process code can be quickly merged into the master to reduce merge conflicts, and feature branches are killed more quickly.
Experimentation
Envoy has evolved their release methodology from continuous rollouts to continuous experimentation. Because the Split platform supports both feature flagging and experimentation, Envoy easily augmented their practice to include analytics. There was no need for teams to implement another toolset, or re-enter segment lists.
Product managers at Envoy are laser-focused on achieving aggressive revenue growth goals. Split is helping Envoy achieve business objectives faster than ever before as product managers now launch experiments every two weeks. They like how easy it is to ‘bucket’ customers and track what treatments have been exposed to each customer so they can quickly analyze results. Many of Envoy’s experiments focus on increasing user activations since these drive revenue flow.
Another key business goal for the company is to ensure customers discover the value of Envoy services as fast as possible. The customer onboarding process involves many steps that include setup up of all devices, getting NDAs signed, photos collected, and much more.
The Envoy team developed a new customer education program to help customers get up and running quickly. As components of the program were tested, Envoy tracked their onboarding flow, measuring how long admins took to sign in their first lobby visitor, to validate if the new capabilities did indeed improve customer onboarding time.
Product managers use their splits in creative ways to increase revenue. For example, they are able to split based on license entitlements, making it easy to test new features on a subset of customers in each of their pricing tiers. They run promotions that temporarily turn on a top-tier feature for a customer who is on a lower tiered plan to encourage them to upgrade. This process also helps them build a close relationship with their customer base as they track how new features change behavior, giving them confidence that scaling to 100% will be successful.
Choosing Split
Before deciding on Split, Envoy analyzed multiple feature flagging tools. Split made the most sense to them because of the ability to easily support their requirements across all environments, from unit tests through to production. The majority of product development at Envoy occurs in a phased rollout manner with feature flags, giving the team full control by only exposing features to select users as required, even if code is released directly to production. For more complex functionality, they will segment splits to run only in test environments initially.
Future Usage
As product and development teams increase their Split usage, Envoy continually refines best practices. Because Split integrates with Jira, they are investigating the use of automated Jira ticketing to remind the team to kill a split if it has been sitting at 100% for too long to help reduce complexity when managing a large volume of splits. Envoy is also investigating the use of Split to fully manage user entitlements of all their pricing tiers.
Deliver software features that matter, fast
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Switch on a free account today or schedule a demo to learn more.