Variant Execution (A/B Testing)

Variant execution extends pipelines with A/B testing capabilities, allowing weighted random selection of pipeline variants, quota enforcement, and coverage gap detection.

Key concepts:

• Variant: a named configuration override applied to a pipeline run, typically differing in prompt, model backend, or parameters
• Weights: each variant has a weight controlling its selection probability
• Quota enforcement: concurrent run limits per pipeline apply regardless of variant selection
• Coverage gaps: detects which eval definitions are not covered by each variant

Variant groups are created, managed, and executed via the Variant Groups API. Results can be compared side-by-side in the Variant A/B Testing view.

For the full specification, see PRD §8.19.