Running one A/B test well is manageable. Scaling A/B testing to hundreds of experiments across dozens of teams is harder, because teams may make different decisions about metrics, randomization, rollout, and analysis. Experiment protocols inDatadog Experiments turn an approved methodology into a reusable configuration, giving experiments of the same type a consistent starting point.

In this post, you’ll learn how to:

Turn an approved methodology into a reusable protocol

Turn an approved methodology into a reusable protocol

Define primary metrics and guardrails before launch

Define primary metrics and guardrails before launch

Standardize randomization and staged exposure

Standardize randomization and staged exposure

Validate experiment data before interpreting results

Validate experiment data before interpreting results

Adapt protocols to recurring experiment types

Adapt protocols to recurring experiment types

Turn an approved methodology into a reusable protocol

Most experimentation programs already agree on how a recurring test should run. The agreement sits in a review document, and every experimenter rebuilds it by hand. Protocols move that agreement into the platform, where an owner defines the configuration every experimenter starts from.

A protocol carries the subject type, primary and secondary metrics, randomization defaults, exposure schedule, duration, analysis settings, and notification recipients. Every protocol is in one of three states: draft, published, or archived. Only published protocols appear when an experimenter creates an experiment. Owners can duplicate an existing protocol rather than build a new one from scratch.

During experiment creation, the experimenter selects a published protocol and Datadog copies its values into the draft. Copied values stay editable, so an experimenter can adjust the configuration as the hypothesis requires. The draft keeps a reference to the protocol it came from, which tells a reviewer what standard the setup follows. Later configuration edits apply to new experiments and leave running experiments unchanged. Changes to a protocol, including its name or description, apply only to new experiments.

Define primary metrics and guardrails before launch

Discovering missing instrumentation after 3 weeks of testing can leave you without the evidence needed for a decision. Define every metric you plan to report on in the pre-launch experiment draft. A protocol makes that the default, since the primary metric, secondary metrics, and guardrails arrive prefilled from a configuration the team already approved.

Datadog Experiments uses two kinds of decision metrics. A primary metric measures the main outcome, and Datadog uses it to determine whether an experiment succeeds. Secondary metrics act as guardrails, checking for unintended harm to areas such as performance, engagement, or revenue. A metric can be primary or secondary, but not both. The primary metric does not decide on its own, because a guardrail regression can still block a rollout.

Each metric has a type of simple, ratio, or percentile, plus the direction you expect it to move. The expected direction is what lets Datadog flag a guardrail moving the wrong way without anyone watching for it. Metrics draw from Datadog’s client SDKs or from a customer warehouse table.

Match the observation window to the outcome

For a checkout experiment, use checkout completion as the primary metric and application errors and page performance as guardrails. Add warehouse metrics to measure business outcomes over the relevant observation window. For example, a retailer whose product returns mostly occur within 2 weeks could measure returns over that period.

Customer lifetime value (LTV), however, may require months of observation. If that exceeds the decision window, choose a shorter-term indicator and validate its relationship to LTV. Warehouse data can support longer-term measurement; the observation window should reflect the outcome and decision being studied.

Set acceptable guardrail thresholds before launch. A regression that materially harms users or the business should stop the rollout even if the primary metric improves. For smaller changes, assess the effect size, uncertainty, and remediation plan before deciding whether to proceed.

Standardize randomization and staged exposure

Set the randomization unit for each protocol

Use a consistent assignment method across the services that participate in an experiment. One deterministic approach hashes the subject identifier with a salt unique to the experiment, meaning that each experiment’s assignments are statistically independent. The salt is an additional input that separates assignments across experiments. When the inputs and assignment configuration stay unchanged, each implementation can compute the same variant without storing a separate assignment. Use the same hash algorithm, input encoding, salt, and allocation rules across languages and frameworks. Continue recording exposure events for analysis even when variant assignment is computed deterministically.

Choose a subject identifier that remains stable for the experience you are testing. For a test before sign-in, this might be an anonymous identifier stored in a cookie. For a test after sign-in, it might be a persistent user ID. Define how identity is handled at sign-in so assignment records and metric events remain associated with the intended subject.

In Datadog, thesubject type determines how exposures join to metric events. For experiments using Datadog Feature Flags, pass a matching identifier as the SDK’s targetingKey. If you change a protocol’s subject type, Datadog asks you to confirm removal of incompatible metrics. Choose the randomization unit for the hypothesis and data available to the team.

Stage exposure and watch guardrails at every step

Protocols configure staged exposure with the same rollout controls as an ordinary experiment setup. A rollout of 1%, 5%, 25%, and then an even split limits how much damage a bad change does before you see it. If an experiment does not need a staged rollout, set its target exposure directly.

Watch request latency and frontend performance at every stage, plus the behavioral signals the owning team depends on. A checkout team watches add-to-cart conversion and payment success throughout the rollout rather than only at the end.

Set a default duration that spans at least one full business cycle. Early spikes and dips often reflect novelty or change aversion rather than the long-run effect. The protocol applies that duration across experiments in the same class.

Validate experiment data before interpreting results

Datadog runs diagnostic checks alongside experiment analysis, covering traffic imbalance, missing assignments, mixed assignments, dimensional assignment imbalance, missing metric data, and pre-experiment metric imbalance. Results appear as a warning, a metric message, or a failed diagnostic check, and those categories carry different weight. Start with any failed check or warning banner before interpreting lift, confidence intervals, or global lift.

Traffic imbalance, also called sample ratio mismatch (SRM), checks whether the observed assignment counts across variants differ significantly from the configured traffic split. Treat a failure as a blocker, because it means the randomization, exposure logging, or analysis population is biased. Dimensional assignment imbalance goes a layer deeper and tests whether assignment probability differs across dimension values such as device type or country. Say you expect an even split and see 48/52. A dimensional failure on device type can localize the cause, such as a device-specific bug in variant delivery or exposure telemetry.

Mixed assignments catch the same subject appearing in more than one variant, and Datadog excludes those subjects from analysis. The diagnostic passes at 5% or less, warns above that, and fails at 20% or more. A rising excluded share usually points at an instrumentation or targeting change rather than a result worth reading.

Check for harm within segments

An overall improvement can hide harm within a segment. Datadog’ssegment-level degradation diagnostic warns when a segment moves against the desired metric direction and performs significantly worse than comparable segments. Investigate a latency increase on one device type or a conversion drop in one region before expanding exposure.

Standardize the analysis plan

Agree on the statistical analysis plan before launch. Set the confidence interval method and confidence level, then choose whether to apply multiple testing correction. Also review controlled-experiment using pre-experiment data (CUPED), a technique that reduces metric variance. Reusing these settings gives teams a consistent basis for interpreting similar experiments.

Theanalysis method determines how interim results can inform a decision. Sequential frequentist analysis supports repeated checks and early decisions while controlling the false positive rate. For fixed-sample analysis, choose the sample size or duration in advance and avoid changing it based on interim results. Bayesian analysis combines observed data with a prior and requires its own decision criteria.

Use pre-experiment data to reduce variance

CUPED uses information collected before exposure to reduce metric variance. It is enabled by default and can be changed in the statistical analysis plan. Collect representative pre-experiment data to support the adjustment. The benefit depends on how informative that data is about the measured outcome. CUPED does not repair biased randomization.

Adapt protocols to recurring experiment types

Create separate protocols when recurring experiments require different designs or decision processes. A single checkout protocol can combine conversion metrics, performance guardrails, and warehouse business outcomes. A warehouse data source alone is not a reason to create another protocol.

Consider three recurring use cases: feature rollouts, UX experiments, and ML and AI experiments. Release teams may prioritize repeatable setup using feature flags within their development workflow. UX teams need results that engineers, product managers, and data scientists can investigate. ML and AI teams may need to compare configuration variants and plan for interactions among experiments.

The following examples illustrate how protocol defaults can differ by use case:

Document additional design requirements

For configuration experiments, also document schema validation for JSON-valued flags, mutually exclusive experiment groups, and holdout populations kept unexposed to a change. Record these requirements alongside the protocol and implement them in the relevant flag or experiment configuration. Add a protocol when a new experiment design recurs, and archive protocols that no longer reflect the approved methodology.

Get started with Experiments protocols

A/B testing best practices hold up at scale when the practices are encoded in the experiment configuration itself. Protocols reduce repeated setup and keep metrics, randomization, and analysis consistent across teams, while automated checks flag experiments whose data cannot support a decision. As an experimentation program grows, a small set of protocols keeps new experiments aligned with the methodology the organization already approved.

To get started with Experiments protocols, read our documentation. If you’re not already a Datadog customer,