With guardrail metrics, you can discover that rather than improving things, doing one experiment negatively influences another area of the business.
Guardrail Metrics Keep You on the Right Track
Guardrail metrics keep organizations and product teams on the right track and are used as a tool to ensure that what you’re doing is aligned with your business goals and objectives. They define boundaries and help guide the decision-making process.
Guardrail metrics influence the decision-making process by:
- Defining the range of behavior that is acceptable
- Expediting the decision making process
- Providing the independence to make decisions based on the situation and context, but not sacrifice alignment
- Reducing risk
They define a range of potential decisions. At the same time, they constrain behavior and empower individuals to make decisions within a spectrum and with less oversight. Guardrails make it easier to make decisions faster while ensuring that decisions are aligned with the goals and values of a business.
The guardrail metrics should not degrade in pursuit of whatever metrics are being measured as part of a test. If they do, your team should be notified to take action. Guardrail metrics are meant to be directional and sensitive, yet indirectly aligned to business value. Page load times, app crashes, and unsubscribe rates are an example of this.
Types of Guardrail Metrics
Guardrail metrics can empower you to make an informed decision. When an experiment moves a guardrail metric unexpectedly, then you may want to not put as much trust into the results or you may decide that it’s better to stop your experiment if it’s detrimental to the business or its users. You may even decide to roll back the new feature being released, as the benefit does not outweigh the detrimental impact.
Choosing the Right Guardrail Metrics
Choosing guardrail metrics that suit your organizational structures can be challenging, but they are critical to an organization. To be considered successful, a feature doesn’t necessarily have to improve a guardrail metric.
At the same time, for a change to be successful there should be little to no negative impact on the guardrail metrics. Guardrails that are the most useful, often fall under one of the following categories:
- Financial metrics: These metrics represent business performance, i.e. revenue.
- User experience metrics: These metrics relate to the measurement of the user experience by observing customer engagement with your product. Examples can include return rate, bounce rate, open rate, clicks, or order rate.
- Strategic priority metrics: These metrics may shift over time. For example, during one quarter the priority may be to increase website traffic, but the following quarter the priority is website downloads.
It’s also important to note that more isn’t always better. You don’t want to guardrail every metric. You want to consider that there’s a tradeoff between how many and how thoroughly metrics are protected. At the same time, you’ll also want to consider any potential friction to the product development process.
At the end of the day, guardrail metrics are very useful as a way of checking that you’re improving areas of your business, and not negatively affecting it.
