Model Backends

Model backends provide a common interface for routing LLM calls across different providers. Each agent in a pipeline selects a model backend by name.

Supported backends

Anthropic

{
  "type": "anthropic",
  "config": {
    "api_key": "sk-ant-...",
    "default_model": "claude-sonnet-4-20250514"
  }
}

Supports all Claude models via the Messages API. Recommended for structured pipeline tasks and agent reasoning.

OpenAI

{
  "type": "openai",
  "config": {
    "api_key": "sk-proj-...",
    "default_model": "gpt-4o"
  }
}

Supports GPT-4o, GPT-4o-mini, o-series reasoning models. Good for code generation and classification tasks.

Local models

Connect to any OpenAI-compatible local endpoint:

{
  "type": "openai",
  "config": {
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama",
    "default_model": "llama3"
  }
}

Works with Ollama, vLLM, LocalAI, and any OpenAI-compatible server.

Stub (for testing)

A mock backend that returns configurable preset responses. Use in unit tests and CI only – never in production.

{
  "type": "stub",
  "config": {
    "responses": {
      "default": "This is a stub response."
    }
  }
}

Health checks

Modulo health-checks each backend on a configurable interval and automatically rotates unhealthy ones out of the pool. The status dashboard shows live health per backend.

Per-node selection

Each agent node in a pipeline can specify a model_backend to override the pipeline default. This lets you route different steps to different providers.

Failover

When a model backend is unhealthy, Modulo automatically fails over to a configured fallback backend. Set fallback_backend_ids on any backend to define the failover order. If the primary and all fallbacks are unhealthy, the run fails with a BackendUnavailableError.

Failover emits a model_failover audit event with the primary and fallback backend IDs for observability.