Variant A/B Testing

A/B testing for pipelines using weighted variant groups. Create variant groups with weighted variants, run them against the same pipeline, and compare outcomes (eval scores, token costs, prompt version hashes).

Overview

Variant A/B testing allows you to:

• Define variant groups with weighted selection strategies
• Run individual variants by weight distribution
• Detect eval coverage gaps per variant
• Compare prompt version hashes across variant snapshots

API

Variant CRUD and execution endpoints are documented in the Variant Groups page.

PRD Behaviour Reference

See PRD §8.19 for the full specification of:

• Comparison view UI (eval scores per node per variant, token cost, HITL outcomes, output diff)
• Eval coverage signal (“Variants diverged but evals did not differentiate”)
• HITL partial completion for variant groups
• Pre-eval degraded mode banner
• Prompt version comparison via run_context_overrides