Feature Delivery Glossary

A/A tests can help you to ensure that your A/B testing process is working properly and your A/B test results are telling you exactly what you think they are.

A/B testing can be very similar, with only a change in button color, or very different, with a total change in the way a feature behaves.

Despite these additional variations, though, A/B/n testing works the same way as standard A/B testing.

Application versions refer to distinct releases or iterations of a software application, each representing a specific stage of development or enhancement. These versions typically feature updates, bug fixes, and new features, and are identified by unique version numbers or names.

Bayesian statistics employs Bayesian probability theory to model and update uncertainties about hypotheses. It involves combining prior beliefs with new evidence, using Bayes' theorem, to obtain updated and more informed probability distributions.

Blue/green deployment is a process that reduces downtime and risk by maintaining two clones of production, called blue and green.

A canary deployment, or canary release, is a deployment pattern that allows you to roll out new code/features to a subset of users as an initial test.

A change advisory board is a collective of representatives from different departments within a company who runs formal change management processes.

Chaos engineering, also known as chaos testing, provides a method and tool-set to deliberately introduce failures and outages in a system.

CI/CD is the acronym in software development for the combination of continuous integration (CI) and continuous delivery (CD).

Client-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed-bandit testing occurring in the user’s browser.

The choice of a cloud deployment model is a strategic decision that influences an organization's flexibility, scalability, costs, and security posture. Whether opting for a public, private, hybrid, or community cloud, it is essential to align the chosen model.

A way to avoid the issues caused by configuration drift is to set up feature flags to test your code in production safely.

By incorporating configuration flags into their development workflow, teams can improve their deployment strategies, enhance testing capabilities, and deliver a better user experience. However, it's important to manage these flags carefully to avoid complexity.

Continuous delivery is a software delivery process that allows devs to release software updates to the production environment and end users at any time.

When implementing continuous delivery, shifting toward purpose-built continuous delivery tools is critical. Here, we discuss the most popular CD tools.

Continuous delivery and continuous deployment are two main terms for how features are deployed to production for testing, experimentation, and release.

Continuous deployment is the practice of automatically promoting code changes to prod after they pass all automated tests in a continuous delivery pipeline.

Continuous integration happens when you automate the process of integrating your code to master to ensure there are no conflicts.

A controlled rollout allows the release of new features gradually, ensuring a good user experience before releasing them to larger groups.

This glossary will discuss some of the most commonly-used customer experience metrics and explain the benefits and drawbacks of each.

Every product manager dark launches (or should). In a dark launch, features are released to a subset of your users to enable testing before a wider deployment.

A data pipeline automates the flow of data from one point to another. Defining how data is collected, and in what schema it should be collected.

Database migration is the process of transferring data from one database system to another, often involving changes in database schema, platform, or storage format while ensuring data integrity and consistency.

Deployment automation enables teams to deploy faster, more frequently, and with greater confidence. By automating the deployment pipeline, organizations can achieve more predictable outcomes, reduce the risk of errors, and free up valuable resources to focus on developing new features.

Do no harm metrics are metrics that team's use to ensure nothing bad is happening to your teams metrics due to a feature rollout.

One observable benefit of edge computing is an improvement in page loading time, but actually any type of internet communication can benefit.

An event stream is a series of data points that flow into or out of a system continuously, rather than in batches.

In various contexts such as programming, design, or decision-making, fallback values serve as backups or substitutes to ensure continuity or functionality when the preferred option is not accessible or applicable.

False Discovery Rate (FDR) is a measure of accuracy when multiple hypotheses are being tested at once. Ex: multiple metrics measured in the same experiment.

A false negative result would indicate that “the change being tested has not improved the key metric significantly when in fact, the change generally has a positive impact on the underlying behavior.”

What is a false positive rate, and how is it calculated? How does it compare to other measures of test accuracy, like sensitivity and specificity?

A feature branch is a copy of the main codebase, where an individual or team of software developers work on a new feature until it is complete.

By adhering to feature development best practices and staying user-focused, development teams can successfully introduce new features that enhance the software product and provide value to users. Continuous learning, adaptation, and feedback integration are key to refining feature development processes.

With Feature experimentation, when anyone can run experiments and test product features, this creates a culture of experimentation, inspiring everyone from all teams to test their ideas and gather data.

Feature flag management systems are the infrastructure for implementing feature flags, also known as feature toggles. They can be built in-house or purchased. This glossary introduces the functionality that effective feature flag management systems deliver to software development organizations.

A feature flag is a software development tool used to safely activate or deactivate features without modifying the source code or performing a new deployment. They decouple deploy from release, which enables trunk-based development, testing in production and progressive delivery.

A feature flags framework is a tool for software development that allows individual features of a software product to be individually enabled or disabled.

Feature flags have been instrumental in progressing the DevOps philosophy to achieve continuous delivery of new, high-quality features.

Feature management is informed by the principles of agile and DevOps, where smaller work units are delivered more often.

A feature rollout plan allows the introduction of a set of new features to a group of your user base, often to a limited user set initially.

Feature toggles let developers “toggle” features on and off without releasing new code. Often used as an alternative to feature branching, toggles have a variety of use cases.

Fixed Horizon is a statistical testing method where sample sizes and your experiment goals are defined in advance.

Flag dependencies are criteria that must be met before a particular flag can be set or activated within a system, software, or process. These dependencies are often utilized in software development to control the flow of actions.

An Auto-Kill Webhook is designed to automatically terminate or disable certain processes, tasks, or resources when triggered by predefined conditions. This functionality is used for managing and controlling system resources, enforcing security measures, or responding to errors in real-time.

Guardrail metrics are business metrics designed to indirectly measure business value and provide alerts about any potentially misleading or erroneous results and analysis.

Hypothesis driven development is the scientific method applied to software development.

Using Jira can enhance the feature release process even further, providing real-time feature status from ideation to code deployment.

With a kill switch, when a feature breaks in production, you can turn it off immediately while your team analyzes the issue.

The main branch is the original thread of code for a project’s codebase.

A merge conflict occurs in version control when changes made in different branches of code cannot be automatically merged due to conflicting modifications. Manual intervention is required to reconcile the differences and successfully integrate the changes into the codebase.

A/B testing for mobile apps is about as similar to standard A/B testing as mobile app development is to standard software development.

Mock testing is created to replicate the behavior of real components, enabling controlled testing of individual units and their interactions.

A modern technology stack integrates various technologies and practices to address the complete lifecycle of an application, from development to deployment and maintenance. By carefully selecting and managing these technologies, organizations can build applications that meet current needs.

Monthly Active Users is a KPI that quantifies the number of unique users engaging with a digital product, platform, or service within a specified monthly time frame. It provides insights into user participation, platform popularity, and overall user engagement.

A multi-armed bandit is a problem to which limited resources need to be allocated between multiple options, and the benefits of each are not yet fully known.

Multivariate testing is a method of experimenting with different variations of elements in a feature to discover which variations will drive user behavior.

With the unknowns of your software's failure modes, you want to be able to figure out what's going on just by looking at the outputs: you want observability.

Observability tools are software solutions designed to monitor, analyze, and provide insights into the performance, health, and behavior of systems and applications.

Power analysis is the process of estimating how many users you will need in order to detect an effect of a given size, or how small an effect you can detect.

The production environment serves as the platform where users interact with the product in real-world scenarios. Its management is a critical aspect of IT operations, demanding attention to detail, rigorous planning, and adherence to best practices to ensure reliability.

Progressive delivery uses feature flags to increase speed and decrease deployment risk, and uses a gradual process for rollout and ownership.

A pull request is when you are requesting for someone in your org, possibly a senior engineer, to look at your code, review it, and pull the changes in.

Release management is the strategic process of planning, coordinating, and controlling the deployment of software releases, ensuring they are delivered efficiently, reliably, and with minimal disruption to users or systems.

Release testing refers to the phase in software development where a software product undergoes thorough testing to ensure its readiness for deployment or release to end-users. This process validates that the software meets quality standards and functions as intended.

A rolling deployment is a software deployment strategy that updates components or instances of a system incrementally, one at a time, to ensure continuous availability and minimal downtime. This approach allows the system to remain operational throughout the update process.

Segments are used to organize targeted users into groups to facilitate setting up incremental product releases or experimentation.

With Sequential Testing, you can confidently make informed decisions about your releases, faster since results are always valid.

Server-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed bandit testing occurring on the web server.

When finding out if an admissions program is biased, a surgery is more successful, or an A/B test variant is superior, Simpson’s Paradox may come into play.

Smoke testing is a type of regression test which ensures that your most important, critical functional flows work as intended.

The software delivery lifecycle (SDLC) encompasses the entire process of planning, creating, testing, deploying, and maintaining software applications or systems. It involves stages such as requirements gathering, design, development, testing, deployment, and ongoing maintenance.

Staging refers to the process of deploying and testing software changes in an environment that closely resembles the production environment but is separate from it. This allows developers and testers to validate changes, assess their impact, and identify potential issues.

Getting a statistically significant result means that the result is likely to be the result of a legitimate, useful trend, instead of random data noise.

A t-test is a type of hypothesis test which assumes the test statistic follows the t-distribution. It is determined if there is a statistically significant difference between two groups.

Targeting rules are predefined criteria or conditions used to direct specific actions or resources towards particular targets within a system. These rules help determine how software functionality, such as advertising, content delivery, or user interactions, should behave.

Technical debt is a metaphorical concept in software development that refers to the cumulative cost of choosing expedient, suboptimal solutions during the coding process. These shortcuts may lead to increased complexities, maintenance challenges, and future inefficiencies.

A test environment is a controlled setup or system used to conduct software testing, providing a simulated platform where software or systems can be evaluated under specific conditions to assess functionality, performance, and reliability before deployment.

To test in production means continuing to test features after they have been deployed to production by using staging to test your features.

A treatment is the version of the UI (user interface) that the user sees. You can set a treatment in the UI to be on or off.

Trunk-based development is a software development model prioritizing continuous integration by committing to a central branch, aiding continuous delivery used by experienced teams.

A type I error is a type of statistical error where the test gives a false positive result, when a perfect test would report a negative. It is one of four possible results from a hypothesis test.

A type II error is one of two statistical errors that can result from a hypothesis test.

A key part of the software development process, usability testing provides invaluable feedback on the user experience of a product.

Winning variation refers to the version of an asset that outperforms other variations in achieving a specific goal, such as higher click-through rates, increased conversions, or greater user engagement. It is identified through statistical analysis of experimental data.

The systematic organization, coordination, and optimization of tasks, processes, and activities within an organization to achieve efficiency, collaboration, and streamlined productivity.