Variant Groups

A/B test variant management – named sets of runs against the same pipeline that differ only in run_context_overrides.

API Endpoints

POST /api/v1/variant-groups – Create a variant group with weighted variants
GET /api/v1/variant-groups – List variant groups (paginated, filterable by pipeline_id)
GET /api/v1/variant-groups/{id} – Get variant group by ID
PUT /api/v1/variant-groups/{id} – Update variant group
DELETE /api/v1/variant-groups/{id} – Delete variant group
POST /api/v1/variant-groups/{id}/run – Run a single variant (weighted random selection)
GET /api/v1/variant-groups/{id}/coverage-gaps – Detect eval coverage gaps per variant
GET /api/v1/variant-groups/{id}/prompt-diffs – Compare prompt version hashes across variant snapshots

Weighted random variant selection with configurable weights per variant
Coverage gap detection showing which eval definitions aren’t covered by each variant
Prompt diff comparison across variant snapshots
Concurrent run quota enforcement per pipeline
RLS org context on every endpoint