Skip to main content
← All posts

Why we version everything (and what happens when you don't)

Modulo Team

versioningauditgovernance

Every team versions its source code in git. Few version their CI pipeline configuration, deployment scripts, model settings, prompt templates, or gate thresholds, so when something breaks they cannot reproduce the failure: they do not know what version of the pipeline ran.

What happens when you do not version

A run fails in a way you have never seen, and you need three answers: what version of the pipeline ran, what version of each component was used, and what changed since the last successful run. If the configuration is mutable, you cannot answer any of them: the pipeline that ran is not the one that exists now, so you debug against a ghost. This is the normal state of most agent pipelines - prompts edited in place, model backends swapped from a dropdown, gate thresholds tuned by sliding a number, every change invisible until something breaks.

What Modulo does differently

Modulo versions everything that affects a run's outcome. When a run starts, we freeze the pipeline definition into a snapshot: every agent version, schema pin, connector binding, model backend pin, prompt version hash, and environment profile. Nothing resolves live; the run executes against this frozen state.

  • You can inspect any past run and see exactly what configuration it used.
  • You can diff two runs and see what changed between them.
  • You can reproduce the run's exact configuration and re-execute it against the same inputs: same agent version, schema pin, model backend, and prompt hash. The LLM's output is not reproducible - models are stochastic, so a re-run may produce different text - but the context of a failure always is.
  • You can change any component without affecting in-flight runs.

The snapshot is not optional; it is how the system works.

What this means for audit and governance

An audit trail that records what happened but not what configuration produced it is incomplete. Knowing a gate failed is useful; knowing the threshold changed thirty minutes before tells you whether it was a real problem or a configuration mistake. Every run carries its own context, so you never have to ask what version of the pipeline was running.

The principle behind this: immutability of runs - a run executes against a frozen snapshot, so the run you inspect is the run that happened.

For the full list of versioned entities, see all first-class versioned entities in Modulo.