Variants run the same pipeline input through different configurations and compare the results. Test two prompts against each other, swap models on a single node, or try different output schemas, all without changing the pipeline itself. The comparison view shows eval scores, token costs, and output diffs side by side, so you pick the configuration that performs better instead of guessing.

In short

What it does: a variant group splits a single pipeline node into weighted alternatives. Each run picks one variant, and the comparison view shows how they performed against the same input.
When you would use it: deciding between prompt phrasings, choosing which model handles a step, or validating a schema change, where you need evidence instead of a hunch. Use it when the cost of a wrong choice is meaningful, such as a high-volume node where a 10% quality improvement compounds across thousands of runs.
When you would not: permanent A/B traffic splitting for end users, or trivial choices where the difference is obvious and the stakes are low. Variants tune your pipeline configuration, not live audience experiments.

The problem it solves

Say you run a pipeline that generates support replies. The current prompt works most of the time, but a rephrasing might reduce vague answers. Without a comparison mechanism, you argue from cherry-picked examples, nobody can point to data, and the change gets blocked or shipped without evidence. The same problem appears when evaluating models: one teammate swears by a smaller, cheaper model for classification, another insists the larger model catches edge cases. Both are right about specific runs, but neither can prove it across a representative sample. You need to run the same inputs through both options and look at the numbers.

How it works

Variant groups

A variant group is a named set of weighted alternatives attached to a single pipeline node. You add variants with their configurations and set weights that control how often each runs: a variant with weight 3 runs three times as often as one with weight 1.

{
  "variant_group_id": "reply-tone",
  "variants": [
    { "id": "v1", "weight": 1, "prompt": "Be concise and direct..." },
    { "id": "v2", "weight": 1, "prompt": "Be warm and reassuring..." }
  ]
}

Both variants run the same pipeline input. The only differences are the configurations you are testing: a prompt, a model backend, a schema, or a combination.

Variant execution

When a run hits a node with a variant group, the system selects one variant by weight and executes that configuration. Every run picks exactly one variant, so each result belongs to a specific configuration. Selection is deterministic per run: replaying a run produces the same assignment, and you can trace an output back to the configuration that produced it.

Comparison views and evals

The comparison view shows a side-by-side breakdown of each variant's performance: eval scores per node, token costs, human-in-the-loop outcomes, and output diffs. If one variant scores consistently higher, keep it; if one costs three times more in tokens but scores the same, you have your answer on cost efficiency. The view also flags eval coverage gaps: if two variants produce meaningfully different outputs but your evals do not differentiate between them, add or refine your evals. A pipeline that runs variants without evals is collecting data it cannot use.

Variants generate the alternatives; evals measure which is better. Without evals, the comparison view shows raw outputs and token costs; with them, it shows which configuration produces higher quality for your use case. If you are not using evals yet, variants still help you compare costs and spot obvious output differences.

Journey example: before and after

Before

A team lead named Sam manages a pipeline that generates technical documentation from code comments. The current prompt produces accurate but dense paragraphs, and Sam wants to try a version that breaks them into shorter sections with headings. The only way to compare is to change the prompt, run a few manual tests, and make a call on gut feeling. Sam ships the new version, and two weeks later the team finds it is longer and increases token costs by 40% without any measurable improvement in reader engagement. There was no baseline, so the decision was made in the dark.

After

Sam sets up a variant group on the documentation node with two variants, the original prompt and the new formatting prompt, each at weight 1. Over the next week the pipeline runs both against the same codebase inputs. The comparison view shows the new prompt scores 3% higher on readability but costs 42% more in tokens per run, so Sam keeps the original. Two weeks later a third variant that shortens paragraphs without headings scores 5% higher on readability and costs only 8% more; that one ships.

Where it fits in a pipeline

Variants sit alongside gates and evals in the pipeline governance stack: gates control when work ships, evals measure quality, and variants help you pick the best configuration before you commit. For the gate types that protect each step, see How HITL gates stop bad agent output before it ships. For the principles behind Modulo's approach to pipeline design, see The principles of Modulo.

Go deeper

Read the full Variant A/B Testing docs for the complete API reference and comparison view details. Learn about evals to set up scoring for your variant outputs. Take the product tour to see the comparison view in action, or use the contact page to talk to us about setting up variants in your own pipeline.