Vistaprint Corporate Solutions (VCS) is a division of Vistaprint, the online print provider that is known and loved by small businesses. Adrienne Dion, Sr. Product Manager, describes Vistaprint Corporate Solutions as an offering for corporate customers that makes it “ridiculously easy for companies to order and manage their print and promotional products”. VCS provides larger businesses a ProShop; a customized storefront, that allows their employees to order company branded business cards, brochures, t-shirts, signs, and more.
Behind the ProShop is a team of 25 engineers, 2 product managers, and a site analyst. Adrienne, along with Ramon Armen, Development Team Lead, develop the additional features and functionality required by the ProShop’s customers that need to scale up their merchandise design and purchases.
“90% of our changes are behind a split (feature flag).”
Ramon Armen, Development Team Lead
Accelerating Growth Through Development Efficiency
In 2019, VCS was looking for accelerants to grow their business. In practice, this meant better capabilities to roll out new features to customers and a scalable way to A/B test which enhancements were most valuable for customers.
Up to this point, VCS had a homegrown system for feature rollouts. It was built to enable early access to new features for select accounts. But this system had a number of drawbacks: it didn’t control access through the entire development process, as feature flags do. It didn’t have sophisticated targeting for users in specific geographies or with certain titles. It was labor intensive to manage and coordinate, so product team members weren’t able to use it. And it didn’t include the ability to measure impact or calculate experimental results based on early feature access.
When Ramon and Adrienne were looking at solutions they chose Split because it accomplished both of their requirements: streamlined feature rollouts and A/B testing. Split was exactly what they needed to turn on and off any feature their 25-person team builds and to kill features that are harming user experience. On top of that, Split pairs feature rollouts with unlimited A/B tests to try out new ideas the team develops. This gave Adrienne the confidence that Split would scale to their needs. In addition to overall capabilities, they selected Split based on ease of implementation with their tech stack and the ease of use for their non-technical product and analytics teams.
The Life (and Death) of a Feature
VCS was quick to get Split set up with their development environment. This involved making Split part of the process during code reviews, ensuring that new features were behind a flag. The team is now up to over 90% of features behind a flag.
A typical feature release involves a percentage rollout, rather than a binary on/off, to test whether a feature works. VCS uses targeting to test features on a portion of users based on their accounts, titles, and location. This targeting can be highly customized: one change involved making users who were eligible for free shipping to be more aware with an in-product banner. The team targeted customers based on their price list to determine eligibility for free shipping and proved that his tactic led to an increase of ~$5 in average order value.
As Ramon describes, they decided to use the .Net server-side SDK, rather than client-side ones, to drive all feature rollouts. The team preferred centralizing the logic for ease of use and maintenance. They also didn’t always want to expose to the client – the end user’s browser or mobile app – the tests that are being run.
Server-side feature flags power feature changes across the application, from backend services to front-end layouts. In one backend use case, the VCS team uses Split to swap out and migrate services. One key aspect of the eCommerce application is image viewing, so as the team was trying out a new image preview service they used Split to release it independently of code deployment and then validate its performance.
Not all features or code changes work out as planned. Adrienne notes that her favorite transformation that Split has brought about is the ability to kill a feature in production.
“The kill button is super easy. Now I just, ‘whoop’, kill it. If we are seeing something wonky, it makes it so easy to turn off. When we’ve triaged the feature we can go back and re-enable it.”
Adrienne Dion, Sr. Product Manager
Monitoring Feature Releases
VCS places a high bar on user experience and site performance. So each new feature or code change needs to be scrutinized to ensure that performance is improving.
In one case, the team had a new feature that would enable users to select their quantity before designing that they had started to roll out to a small percentage of people. Within minutes they were seeing effects they weren’t anticipating. Page load time spiked, error rates increased, and performance issues started cropping up in other downstream services. Split presents these metrics alongside each feature, so the team was able to quickly spot the performance change, identify which feature caused it, and kill the feature instantly. The VCS team was then able to go back and fix the underlying cause without the complexity of a code roll back or the customer experience impacts and time pressure of a hotfix.
In another feature release, Split identified a specific user segment in Europe affected by an issue. The team continued the roll out in North America while turning off the feature in Europe, using targeting on the feature flag. In the meantime they reworked the feature to resolve the issues in Europe and redeployed the change.
Using the .Net SDK to track event data and send it to Split, VCS tracks a set of key metrics for every feature to ensure that user experience improves. These metrics include page load time to measure responsiveness, conversion rate to measure business objectives, and exit rate to measure interruptions to their user journeys.
Using Experimentation to Expand into New Markets
One of VCS’s original goals was to add A/B testing capabilities and bring in a culture of experimentation. Split has helped VCS start down an experimentation journey; they hadn’t done A/B testing before as an organization. Their first experiment was changing the purchase and design path for a new product, adding a new summary view for the product that the user was about to customize. The VCS team hoped providing additional context to the user would lead to a better conversion rate to order. They started with a 50-50% rollout to test the new user journey. And it worked; they saw an increase in conversion rates from studio to checkout.
COVID-19 has sped up this experimentation culture. The team used Split to build out an inventory service in record time. They use Split to control the rollout by country, as additional inventory becomes available to support new markets. And they use Split to control rollouts to individual customer accounts, including or excluding accounts based on their desire or aversion to seeing new features. These capabilities have enabled Vistaprint to move at a rapid pace amid changing conditions for small business, right on goal.
Deliver software features that matter, fast
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Switch on a free account today or schedule a demo to learn more.