Feature Spotlight: How evals score your agent output so you know what shipped

You run an agent, it produces output, and that output moves downstream to a merge, a deploy, or a customer. The question is whether that output was good enough. Evals are automated quality checks that score agent output against rubrics, patterns, or schemas. They run after a node completes and before any human review, so every piece of output is measured against the criteria you set before it reaches the next step.

In short

• What it does: each eval runs a check (LLM judge, regex, schema, or custom function) against the output of a pipeline node and returns a pass or fail.
• When you'd use it: anytime you need to know whether agent output meets a quality bar before it moves downstream.
• When you wouldn't: when the output is trivial, read-only, or already covered by an existing automated gate that catches the same failure modes.
• Failure behaviour is configurable: warn to log and continue, or block to stop the run.

The problem it solves

Say you have an agent that writes code and opens a pull request. It passes the basic structural checks: the diff is well-formed, the branch name is valid, the commit messages follow convention. Then it ships a change that introduces a race condition, or writes a test that always passes regardless of input, or crafts a response that technically answers the customer but misses the real concern.

The structural checks did not catch it because they were never designed to. Schema validation confirms shape, not substance. A regex catches patterns, not intent. You need a different kind of check: one that evaluates the output against a quality standard you define, scores it, and decides whether it is good enough to continue. Without evals, the only quality gate between agent output and the next step is a person reading everything manually, which does not scale.

How it works

Evals are post-node steps. When a pipeline node finishes, its output passes through every eval you have configured before the pipeline moves on. Each eval runs independently, scores the output, and either passes or fails.

LLM judge

An LLM judge sends the agent output to a separate model with a rubric prompt. The model returns a score from 0 to 1 and reasoning. You configure which model backend to use (independent from the agent's own backend), the rubric text, and the pass threshold. Use this when quality is subjective: tone, completeness, persuasiveness, or anything that a regex or schema cannot capture.

Regex

A regex eval applies a pattern to the output text. A match means a pass. There is no numeric threshold. Use this when you need to confirm the presence (or absence) of a specific pattern: a data classification label, a required footer, or the absence of a hardcoded credential.

JSON schema

A JSON schema eval validates the output against a formal schema definition. This is stricter than the agent's own output schema. Use this when you need semantic validation beyond shape: ensuring a list is non-empty, a URL resolves, or a required field matches a format constraint that the base schema does not enforce.

Custom function

A custom function calls a Python function registered via the modulo.evals entry-point group. The function receives the output and returns a score from 0 to 1. Use this when the check involves logic that does not fit the other types: cross-referencing against external data, running a deterministic algorithm, or evaluating a compound condition.

Configuration

Every eval has a pass_threshold (0 to 1) and a failure_behaviour. Set threshold to the minimum score that counts as passing. Set behaviour to warn to log the result and let the run continue, or block to stop the run with an eval_failed terminal state:

{
  "id": "quality-check",
  "name": "Output quality",
  "type": "llm_judge",
  "config": {
    "model_backend_id": "review-model",
    "rubric": "Score the response on completeness (0-1), accuracy (0-1), and clarity (0-1). Weight equally."
  },
  "pass_threshold": 0.8,
  "failure_behaviour": "block"
}

Conditional HITL gating

Evals connect directly to human-in-the-loop gates. A HITL gate can reference an eval in its condition field, creating a gate that only pauses when the eval score falls below a threshold. If the output scores well enough, the pipeline continues without human review. If it does not, the run pauses and a reviewer sees the output, the eval score, and the reasoning. This lets you route only the borderline cases to a person while letting high-confidence output through.

Guardrails as input-side evals

Guardrails are the input-side counterpart to evals. Where evals check output after a node runs, guardrails check input before a run is created. They use the same EvalDefinition structure with type set to guardrail, but their detection is deterministic (regex or JSON schema only) and they run at the ingestion edge. A guardrail can observe, warn, block, or redact the offending content. A block is terminal: the run is never dispatched, never retried, and has no HITL gate.

Journey example: before and after

Before

Priya runs a team that uses an agent to generate release notes from merged pull requests. The agent is fast. It reads the diff, summarizes the change, and posts the notes to a changelog. Most of the time the output is fine. Then one week the agent summarizes a breaking database migration as "minor improvements," and the team publishes it. Customers see "minor improvements" and upgrade. The migration breaks their integrations. The team spends the rest of the week fielding support tickets and rolling back.

The problem was not that the agent wrote bad notes. It was that nothing measured the output against a quality standard before it reached the publish step. The schema check confirmed the notes were well-formed JSON. It did not confirm the notes were accurate.

After

Priya adds two evals. The first is an LLM judge with a rubric that scores completeness and accuracy against the diff. The second is a regex eval that checks for the presence of "BREAKING CHANGE" when the diff touches migration files. Both evals run at block. The publish step now only runs when both evals pass. The LLM judge catches misleading summaries. The regex catches missed breaking changes. The team publishes accurate notes, and the support ticket volume drops to zero for that category.

When to use it vs. when not to

Use evals when the quality of agent output matters and the failure mode is something a person would catch by reading: incomplete answers, inaccurate summaries, output that violates a content policy, or responses that miss the point. Use them when you want to automate that judgment and only escalate borderline cases to a person via conditional HITL gating.

Do not add evals to every node. A pipeline with an eval on every step adds latency and cost for output that is trivial or already covered by a structural check. If a schema check confirms the output shape and a regex catches the failure mode you care about, that is enough. Evals are for the checks that require judgment. Reserve them for the steps where quality variation is real and the cost of bad output is worth preventing.

Where it fits in a pipeline

Evals sit between node execution and HITL gates. Automated gates catch the patterns they were written to catch, evals measure quality against the standards you define, and checkpoints catch the judgment calls that need a person. Together they reduce the amount of manual oversight a pipeline needs as you gain confidence in its output. For the gate type that pauses for human review, see How HITL gates stop bad agent output before it ships. For the broader governance model, see The principles of Modulo.

Go deeper

Read the Evals docs for the full configuration reference, or the related docs on HITL, schemas, and agents. Try the hosted demo at demo.modulo.run, or use the contact page to talk to us about putting evals in your own pipeline.