Skip to main content
Modulo

Evals

Evals are automated quality checks on agent output. They run as a post-node step, after node completion and before any human-in-the-loop (HITL) gate check. Each eval definition specifies a type, configuration, pass threshold, and failure behavior.

Eval definition

Each eval definition has:

Eval definition
Field Description
id Unique identifier
name Human-readable name
type llm_judge, regex, json_schema, custom_function
config Type-specific configuration
pass_threshold Numeric threshold between 0 and 1
failure_behaviour warn - log the result, run continues | block - run transitions to eval_failed terminal state

Eval types

Large language model (LLM) judge

Sends agent output to an LLM with a rubric prompt. Returns a score from 0 to 1 and reasoning. Uses a separate, independently configured model_backend_id, not the agent’s own backend.

Regex

Applies a regex pattern to the output. Pass means a match was found. No threshold.

JSON Schema

Validates output against a JSON Schema definition. Stricter than the agent’s output_schema - for catching well-typed but semantically invalid outputs.

Custom function

Calls a Python function registered via the modulo.evals entry-point group. Returns a score from 0 to 1.

Eval result record

Each eval run stores a result per run per node:

Eval result record
Field Description
eval_id The eval definition that ran
run_id The run
node_id The node whose output was evaluated
score Numeric score, nullable for non-scored types
passed Whether the eval passed
detail Human-readable detail
evaluated_at Timestamp

Results are displayed in the run inspection UI. A block failure is written to the AuditEvent log.

Conditional HITL gating

A HITL gate’s condition field can reference an eval result. The three eval fields are nested inside the gate’s condition object:

Pipeline config 
{
  "type": "gate",
  "human_only": true,
  "condition": {
    "eval_id": "quality-check",
    "threshold": 0.8,
    "operator": "lt"
  }
}

If the condition is true (score < 0.8), the pipeline pauses for human review. If false, execution continues without pausing.

Guardrails

Guardrails are the input-side analogue of evals: deterministic data-safety checks at the ingestion boundary.

  • A guardrail is an EvalDefinition with eval_type="guardrail"
  • Detection is deterministic and pure (regex or json_schema only)
  • Interception runs at run-creation, before the input payload is persisted

Actions:

Guardrails
Action Behavior
observe Log only
warn Warn and continue
block Terminal eval_failed (error code eval_blocked)
redact Redact the offending content

A block is terminal: the run is never dispatched, never retried, and has no HITL gate.

Policy-as-code

Guardrail configuration is versioned, hashed YAML. You propose a change, review the diff, and apply it through a pull request. Each run’s snapshot pins the guardrail configuration it ran under, with drift detection.

Next: Library