Evals

Evals are automated quality checks on agent output. They run as a post-node step, after node completion and before any human-in-the-loop (HITL) gate check.

Eval definition

Each eval definition has:

Eval definition
Field	Description
`id`	Unique identifier
`name`	Human-readable name
`type`	`llm_judge`, `regex`, `json_schema`, `custom_function`
`config`	Type-specific configuration
`pass_threshold`	Numeric threshold between 0 and 1
`failure_behaviour`	`warn` - log the result, run continues \| `block` - run transitions to `eval_failed` terminal state

Eval types

LLM judge

Sends agent output to a model with a rubric prompt. Returns a score from 0 to 1 and reasoning. Uses a separate, independently configured model_backend_id - not the agent’s own backend.

Regex

Applies a regex pattern to the output. Pass means a match was found. No threshold.

JSON schema

Validates output against a JSON Schema definition. Stricter than the agent’s output_schema - for catching well-typed but semantically invalid outputs.

Custom function

Calls a Python function registered via the modulo.evals entry-point group. Returns a score from 0 to 1.

Eval result record

Each eval run stores a result per run per node:

Eval result record
Field	Description
`eval_id`	The eval definition that ran
`run_id`	The run
`node_id`	The node whose output was evaluated
`score`	Numeric score, nullable for non-scored types
`passed`	Whether the eval passed
`detail`	Human-readable detail
`evaluated_at`	Timestamp

Results are displayed in the run inspection UI. A block failure is written to the AuditEvent log.

Conditional HITL gating

A HITL gate’s condition field can reference an eval:

Pipeline config

{
  "eval_id": "quality-check",
  "threshold": 0.8,
  "operator": "lt"
}

If the condition is true (score < 0.8), the pipeline pauses for human review. If false, execution continues without pausing.

Guardrails

Guardrails are the input-side analogue of evals - deterministic structured-credential boundary data-safety at the ingestion edge.

A guardrail is an EvalDefinition with eval_type="guardrail"
Detection is deterministic and pure (regex or json_schema only)
Interception runs at run-creation, before the input payload is persisted

Actions:

Guardrails
Action	Behavior
`observe`	Log only
`warn`	Warn and continue
`block`	Terminal `eval_failed` (error code `eval_blocked`)
`redact`	Redact the offending content

A block is terminal: the run is never dispatched, never retried, and has no HITL gate.

Next: Library