Feature Spotlight: How A/B testing finds the best agent configuration

Variants let you run the same pipeline input through different configurations and compare the results. You can test two prompts against each other, swap models on a single node, or try different output schemas, all without changing the pipeline itself. The comparison view shows eval scores, token costs, and output diffs side by side, so you pick the configuration that actually performs better instead of guessing.

In short

• What it does: a variant group splits a single pipeline node into weighted alternatives. Each run picks one variant, and the comparison view shows how they performed against the same input.
• When you would use it: deciding between prompt phrasings, choosing which model handles a step, or validating a schema change, where you need evidence instead of a hunch.
• When you would not use it: permanent A/B traffic splitting for end users. Variants are for tuning your pipeline configuration, not for live audience experiments.
• Evals score each variant's output quality so you can see which configuration wins.

The problem it solves

Say you run a pipeline that generates support replies. The current prompt works most of the time, but you heard that rephrasing the system instructions might reduce the rate of vague or unhelpful answers. You could change the prompt and hope for the best. Or you could run both versions at the same time and compare. Without a comparison mechanism, you end up arguing about which prompt is better based on a handful of cherry-picked examples. The team spends hours debating, nobody can point to data, and the change either gets blocked or gets shipped without evidence.

The same problem shows up when evaluating models. One team member swears by a smaller, cheaper model for classification tasks. Another insists the larger model catches edge cases the smaller one misses. Both are right about specific runs, but neither can prove the point across a representative sample. What you need is a way to run the same inputs through both options and look at the numbers.

How it works

Variants are a pipeline-level feature. Here is how you set them up and what each piece does.

Variant groups

A variant group is a named set of weighted alternatives attached to a single pipeline node. You define the group, add variants with their configurations, and set the weights that control how often each one runs. Weights determine the traffic split: a variant with weight 3 runs three times as often as a variant with weight 1.

{
  "variant_group_id": "reply-tone",
  "variants": [
    { "id": "v1", "weight": 1, "prompt": "Be concise and direct..." },
    { "id": "v2", "weight": 1, "prompt": "Be warm and reassuring..." }
  ]
}

Both variants run the same pipeline input. The only differences are the configurations you are testing: a prompt, a model backend, a schema, or any combination of those.

Variant execution

When a run hits a node with a variant group, the system selects one variant based on the weights and executes that configuration for the run. Every run picks exactly one variant, so each result belongs to a specific configuration. The selection is deterministic per run, which means replaying the same run produces the same variant assignment. This matters for debugging: you can trace a specific output back to the exact configuration that produced it.

Comparison views

The comparison view is where the real work happens. For a given variant group, you see a side-by-side breakdown of each variant's performance: eval scores per node, token costs, human-in-the-loop outcomes, and the actual output diffs. If one variant scores consistently higher on your evals, that is the one to keep. If one variant costs three times more in tokens but scores the same, you have your answer on cost efficiency.

The comparison view also flags eval coverage gaps. If two variants produce meaningfully different outputs but your evals do not differentiate between them, that is a signal to add or refine your evals. A pipeline that runs variants without evals to compare them is collecting data it cannot use.

Relationship to evals

Variants and evals work together but serve different purposes. Variants generate the alternatives. Evals measure which alternative is better. An eval definition runs against a variant's output and returns a score. Without evals, the comparison view shows you the raw outputs and token costs. With evals, it shows you which configuration actually produces higher quality results for your use case. If you are not using evals yet, variants still help you compare costs and spot obvious output differences. Adding evals turns that into a data-driven decision.

Journey example: before and after

Before

A team lead named Sam manages a pipeline that generates technical documentation from code comments. The current prompt produces accurate but dense paragraphs. Sam wants to try a version that breaks long paragraphs into shorter sections with headings, hoping it improves readability. The only way to compare is to change the prompt, run a few manual tests, and make a call based on gut feeling. Sam picks the new version, ships it, and two weeks later the team discovers the new format is longer and increases token costs by 40% without any measurable improvement in reader engagement. There was no baseline to compare against, so the decision was made in the dark.

After

Sam sets up a variant group on the documentation node with two variants: the original prompt and the new formatting prompt, each at weight 1. Over the next week, the pipeline runs both versions against the same codebase inputs. The comparison view shows the new formatting prompt scores 3% higher on the readability eval but costs 42% more in tokens per run. Sam decides the cost is not worth the marginal quality gain and keeps the original prompt. Two weeks later, Sam tries a third variant that shortens paragraphs without adding headings, scores 5% higher on readability, and costs only 8% more. That one ships. Without the comparison data, Sam would have either kept the original forever or shipped the expensive version by accident.

When to use it vs. when not to

Use variants when you are choosing between configurations and need evidence: prompt phrasings, model backends for a specific node, output schemas, or any toggle where the right answer depends on actual performance rather than assumption. Use them when the cost of a wrong choice is meaningful, such as a high-volume node where a 10% quality improvement compounds across thousands of runs.

Do not use variants for permanent traffic splitting to end users. Variants are a pipeline tuning tool, not a live experiment framework. Do not use them for trivial choices where the difference is obvious and the stakes are low. Adding a variant group to every node creates overhead with no payoff. Focus on the decisions where data changes the outcome.

Where it fits in a pipeline

Variants sit alongside gates and evals as part of the pipeline governance stack. Gates control when work ships. Evals measure quality. Variants help you pick the best configuration before you commit to one. For the gate types that protect each step, see How HITL gates stop bad agent output before it ships. For the principles behind Modulo's approach to pipeline design, see The principles of Modulo.

Go deeper

Read the full Variant A/B Testing docs for the complete API reference and comparison view details. Learn about evals to set up scoring for your variant outputs. Try the hosted demo at demo.modulo.run to see the comparison view in action, or use the contact page to talk to us about setting up variants in your own pipeline.