Skip to main content
Modulo

Concurrency and capacity sizing

Modulo limits how many things run at once, at several separate layers. Each pipeline caps its own runs, your organization caps its total, the runner layer caps how many sandboxes may be open, and the worker processes that execute runs have their own throughput and connection budgets. Every layer has a different default, a different way of failing, and a different place where you change it. This page describes each limit, what happens when you cross one, how to tell a healthy queue from a stuck one, and how to size the numbers for your workload.

The mental model: slots and queues

A slot is permission for one unit of concurrent work at one layer. A pipeline slot lets one more run of that pipeline execute. A runner slot lets one more sandbox open on your infrastructure. An organization run slot lets one more run hold a worker.

Modulo meters slots because each executing agent consumes real resources: database connections, Redis connections, sandbox containers, host processor and memory, and money. Uncapped concurrency does not fail gradually. It exhausts the shared resources underneath and takes the whole deployment down with it.

The fact that explains most operator confusion is simple: a queued run consumes no capacity. A run waiting for a slot is a database row in the pending state. It holds no pipeline slot, no runner slot, and no organization slot. It starts when a slot frees and the scheduler picks it up. Queueing a thousand runs costs you a thousand rows, not a thousand units of execution.

Two measurement details follow from that rule, and both surprise people:

  • Runs parked at a human gate are neither queued nor executing. They hold no pipeline slot, but they do count against the organization run cap.
  • Some limits measure demand instead of execution. The variant group quota counts every unfinished run of that pipeline, and each trigger’s own concurrency limit counts every unfinished run of that trigger: queued, running, or waiting for a human, not only the ones executing right now.

Every limit you can meet

Every limit you can meet
Limit What it limits Default When you cross it Where you change it
Pipeline concurrency (max_concurrent_runs on the pipeline) Runs of that pipeline that may execute at once 5 Queues. The run stays pending until a slot frees. No web UI field today. Set it through the API (PATCH /api/v1/pipelines/{pipeline_id}) or modulo apply (pipelines[].max_concurrent_runs).
Pipeline queue timeout (queue_timeout_seconds on the pipeline) How long a capacity-queued run of that pipeline may wait before it is superseded No timeout (null = never) Supersedes the run: it ends rejected with queue.expired. Not a failure and never paged. Pipeline editor toolbar (Queue timeout (s)), API, or modulo apply (pipelines[].queue_timeout_seconds)
Trigger concurrency (max_concurrent_runs on the trigger) Unfinished runs created by that one trigger: queued, running, or waiting for a human 1 Recorded as concurrency_limit_reached. A webhook delivery or Slack app-mention is accepted and the run queues as pending; a cron, polling or agent-signal fire is skipped and waits for its next fire; a webhook replay is refused with HTTP 429. Settings > Triggers (web UI field)
Runner cap (sandbox_concurrency_limit) Sandbox dispatches your organization may hold open at once 4 when the key is unset, see the note below Queues. The dispatch is denied and the run retries when capacity frees. Runners > Concurrency (web UI); API PUT /api/v1/admin/org/sandbox-concurrency
Organization run cap (run_concurrency_limit) Executing, claimed and human-waiting runs across all pipelines No limit (unset) Queues. Dispatch defers the run; a background reconciler re-dispatches it when capacity frees. API only (PUT /api/v1/admin/org/run-concurrency). No web UI today.
Variant group concurrency (max_concurrent_runs on the variant group) All unfinished runs of that pipeline (queued, running, or waiting for a human) when the group starts 5 Refuses the start with HTTP 429. No web UI field. Set it through the API (POST/PUT /api/v1/variant-groups). The variant group builder always creates groups with 10 and does not expose the value.
Trigger backpressure guard New trigger fires while a pipeline’s pending queue is already deep or already old Depth over max(3 x pipeline cap, 5), or oldest pending run older than 60 minutes Refuses the new fire only. It never refuses a run that already exists. Depth follows the pipeline cap. The age threshold is TRIGGER_BACKPRESSURE_MAX_AGE_SECONDS (environment variable).
Worker throughput (the Simple Async Queue (SAQ) worker: SAQ_WORKER_CONCURRENCY and the two pool sizes) How many runs each worker process executes at once, and how many database and Redis connections that work holds 5 concurrent jobs, 20 Redis connections and 65 database connections per worker process Refuses at the infrastructure layer. You see connection errors from the database or Redis instead of a clean product-level wait. Environment variables on the server (restart to apply)
Local execution cap (MODULO_MAX_LOCAL_CONCURRENCY) Agent commands run as local subprocesses on the host at once 2 Queues. Commands wait on an internal semaphore. Environment variable, or Settings > Runtime Config (hot editable by system administrators)
Database storage guard New run creation when the database volume is full Auto-detected: DB_CAPACITY_BYTES if set, otherwise the Postgres volume probed at DB_CAPACITY_PATH. Hard stop at 98 percent (DB_CAPACITY_HARD_STOP_PCT); inert when no source resolves Refuses new runs with HTTP 503 (storage exhausted). Environment variables on the server (restart to apply)

Three notes on that table:

  • The runner cap default is conditional. The default of 4 applies while the runner-capacity gate flag (RUNNER_CAPACITY_GATE_ENABLED) is on. With the flag off, an unset key means no gate at all. An explicitly set value is enforced regardless of the flag: an integer caps every dispatch (0 means deny all) and an explicit null means no gate.
  • The backpressure guard covers webhook, cron and polling fires only. Manual runs, Model Context Protocol (MCP) trigger_pipeline calls, agent signals, Slack app-mentions, and replayed webhook deliveries are never refused by it.
  • Some controls live outside this page. Daily spend limits, request rate limits and guardrails are separate features with their own refusals. See Cost Management and Triggers.

Where each setting is changed

Where each setting is changed
Setting Where it lives Who can change it
Trigger concurrency Web UI form (Settings > Triggers), the API, and modulo apply Anyone who can edit that trigger
Pipeline concurrency API (PATCH /api/v1/pipelines/{pipeline_id}) and modulo apply. No web UI field. Anyone who can edit that pipeline
Pipeline queue timeout Pipeline editor toolbar (Queue timeout (s)), API (PATCH /api/v1/pipelines/{pipeline_id}) and modulo apply Anyone who can edit that pipeline
Variant group concurrency API (POST/PUT /api/v1/variant-groups). No web UI field; the builder hardcodes 10 at creation. Anyone who can edit variant groups
Runner cap and organization run cap Admin API. The runner cap also has Runners > Concurrency. Organization administrators
SAQ_WORKER_CONCURRENCY, SAQ_REDIS_POOL_SIZE, SAQ_WORKER_DB_POOL_SIZE, DB_CAPACITY_*, TRIGGER_BACKPRESSURE_MAX_AGE_SECONDS Environment variables only. They are not Runtime Config keys, so a change needs a restart. Whoever operates the server
MODULO_MAX_LOCAL_CONCURRENCY Environment variable at startup, plus a Runtime Config override that applies without a restart. Runtime Config requires the system.config.manage permission, which only system administrators hold. There is no in-product way to grant that role.

MODULO_MAX_LOCAL_CONCURRENCY is the throughput knob you can edit in the web UI: Settings > Runtime Config lists it, validates the value as an integer, and applies it to work that starts afterwards, with no restart. Runtime Config refuses overrides for boot-only keys such as database URLs and secrets; those stay environment variables and need a restart.

What happens when you hit a limit

Most capacity limits behave the same for every way of starting a run: the run exists, and it waits. Two gates break that pattern: the storage guard refuses creation everywhere the same way, and the fire-level gates, backpressure and trigger concurrency, are where the funnels differ - the table below shows which fires are refused, which are skipped, and which are accepted and queued.

A run that queues because of capacity waits indefinitely by default. There is no instance-wide queued timeout any more: such a run is not failed, and nothing is silently dropped. A pipeline can opt into a bound with queue_timeout_seconds; on expiry the run is superseded (rejected / queue.expired), which is not a failure. The retired capacity_timeout code still resolves for historical rows but is no longer written.

What happens when you hit a limit
How the run starts Pipeline queue is over the backpressure limit Trigger is already at its own concurrency limit
Webhook The endpoint answers HTTP 429 and the sender retries later. Accepted; the run queues as pending and a concurrency_limit_reached event is recorded.
Webhook replay Not gated. Refused with HTTP 429 after a concurrency_limit_reached event is recorded.
Cron No run is created and the fire is consumed, so the work waits for the next scheduled fire. A daily schedule that is backpressured at fire time loses that day. The fire is consumed and the work waits for the next scheduled fire.
Polling The cycle is skipped and the next polling interval evaluates again. The cycle is skipped and the next polling interval evaluates again.
Agent signal Not gated. The signal is skipped and a concurrency_limit_reached event is recorded.
Slack app-mention Not gated. Accepted; the run queues as pending and a concurrency_limit_reached event is recorded.
Manual (web UI or API) Not gated. The run is created and queues like any other. Not gated.
MCP trigger_pipeline Not gated. The run is created and queues like any other. Not gated.

Why is my run waiting, and is my queue healthy?

A run waiting for capacity is normal behavior, not a fault. It starts when a slot frees. What matters is not how many runs are waiting, but whether the oldest waiting run still moves:

  • Healthy: the oldest waiting run eventually starts, and the runs behind it start too. The depth may stay high for a long time while long-running agents hold the slots; that is the cap working as designed, not a fault. Queue depth alone is not a health signal.
  • Stuck: the oldest waiting run’s age keeps growing and nothing in that pipeline starts at all. That needs attention.

Where to look today:

  • Run list: a pending run that is waiting at or above the organization cap carries a “queued” badge. A run that a cap deferred also carries an error tag (capacity.org or capacity.pipeline). A run whose queue timeout expired shows as expired while queued (rejected / queue.expired).
  • Run detail: a pending run shows “Queued - waiting for a free slot (N active / M limit)” when the organization cap is holding it, and “Queued - starting soon” whenever no cap is holding it (including when no organization cap is configured). When a cap deferred it, a “waiting for capacity” panel shows the reason detail. A queue-expired run explains that it waited past the pipeline’s queue timeout and was dropped before it started.
  • Error Dashboard: the scheduler starvation banner lists pipelines whose runs have waited more than 10 minutes, with the pending count and the oldest wait.
  • Analytics: the Capacity and queue panel answers “is the queue draining, and which limit is holding it?”. It shows pending depth, oldest pending age, the drain ratio, org cap use, the binding limit and its headroom, pending runs broken down by cause (pipeline, org run, org sandbox, unmarked), active-run health, a per-pipeline table and an interval series. The same report drives the capacity alerts.
  • Capacity alerts: the notification channel sends an escalating ladder per pipeline - a queue-building notice, a warning when the oldest pending run passes 10 minutes, and a critical when at least three runs are queued with nothing draining - plus a backpressure alert (the pipeline is currently refusing new fires) and an org saturation alert (an org run or sandbox limit fully held with work queued). Alerts auto-clear when the queue drains. They fire whether or not a queue timeout is configured, so they are the escalation path for the wait-forever default.
  • Analytics trends: alongside the panel, the analytics series shows average queue wait, average capacity wait, capacity failures and stalls over time.

Sizing your limits

  • Expect long waits; they are legitimate. With a pipeline cap of 1 and an agent run that takes two hours, the next run waits up to two hours. That is the cap working as designed. If that wait is unacceptable, raise the cap or split the work.
  • Do not size from queue depth. Depth measures demand, not health, and a deep queue behind long-running agents drains steadily.
  • Set the pipeline cap from desired parallelism, limited by what each concurrent run costs in model spend, sandbox hours and database load.
  • Set the runner cap from your hardware. Runners > Concurrency shows a preflight warning when the cap exceeds the engine’s processor or memory.
  • Budget the worker layer before raising product limits. Each worker process runs SAQ_WORKER_CONCURRENCY jobs at once, so total executing runs across the deployment is roughly the number of worker processes multiplied by that number. Check the budgets:
database connections = (worker processes x SAQ_WORKER_DB_POOL_SIZE)
                       + web-layer pools + checkpointer
                       <= PostgreSQL max_connections

Redis connections    = worker processes x SAQ_REDIS_POOL_SIZE
                       + web-layer connections
                       <= Redis maxclients

A worker raises its own pools to at least SAQ_WORKER_CONCURRENCY + 5 when you configure them lower, so a too-small pool corrects itself instead of wedging the worker. The budgets above still have to hold across the whole deployment.

  • Raise the product limits only when the infrastructure has headroom. Otherwise you trade clean product-level waits for connection errors that affect every pipeline on the deployment.
  • Leave the backpressure guard alone. Its depth threshold derives from the pipeline cap, so it scales with the cap you already set. Only the age threshold (TRIGGER_BACKPRESSURE_MAX_AGE_SECONDS) is a tuning knob, and raising it lets trigger fires pile up for longer before they are refused.
  • Set a queue timeout only where a late run is useless. A capacity-queued run waits indefinitely by default, which is right for webhook deliveries and for runs a person asked for. A scheduled pipeline is the exception: give it queue_timeout_seconds equal to its period, so a fire that can no longer be useful drops as a supersession instead of running hours late.

If a queue is not draining

  1. Check that runs start at all. In the run list, does any run of the pipeline move from pending to running? If nothing ever starts, the problem is the worker service, not a limit.
  2. Read the Error Dashboard starvation banner. It names pipelines whose runs have waited over 10 minutes, with the oldest wait.
  3. Check the runner cap on Runners > Concurrency. A stored 0 means deny all. Look for capacity.org failures on runs: they mean an organization cap denied the run, and the run’s error detail says whether it was the sandbox cap (“Org sandbox concurrency limit reached”) or the organization run cap (“Org run concurrency limit reached”).
  4. Check the pipeline’s own concurrency. There is no web UI field for max_concurrent_runs, so read it from the API or from your modulo apply config.
  5. Check the organization run cap. Read run_concurrency_limit from the admin API. It is unset by default, and human-waiting runs count against it.
  6. Check human gates. Runs parked on an undecided human-in-the-loop (HITL) gate hold no pipeline slot but do count against the organization cap, so a backlog of undecided gates can starve new work while every pipeline looks idle. Decide or reject the parked runs to release that capacity.
  7. Check the pipeline’s queue timeout. By default a capacity-queued run waits indefinitely, so a long wait is a symptom of a stuck queue, not a run about to time out. If the pipeline sets queue_timeout_seconds, a run past that window is superseded as queue.expired (status rejected) - the run did not fail, it was dropped before it started.
  8. Check storage. HTTP 503 responses naming storage exhaustion mean the database volume hit the hard stop. Export or purge old runs with run retention, or raise the configured capacity.

Reference: exact definitions

  • Queue depth: how many runs of the pipeline are in pending state with no cancellation requested.
  • Oldest pending age: the time since that oldest pending run was created.
  • Backpressure threshold: the guard refuses a new trigger fire when depth is greater than max(3 x max_concurrent_runs, 5), or when the oldest pending age is greater than TRIGGER_BACKPRESSURE_MAX_AGE_SECONDS (default 3600 seconds).
  • Pipeline slot holders: runs that are executing or claimed (running, claimed, unknown). Pending runs and runs waiting for a human (awaiting_human, hitl_parked) do not hold a pipeline slot.
  • Organization run cap population: executing, claimed and human-waiting runs. Pending runs are not counted.
  • Runner cap population: runs that hold a live sandbox dispatch marker. An explicitly configured value counts every provider; the default of 4 counts host-backed dispatches (Docker and local) only.
  • Slot saturation: dispatch also defers a new admission once a pipeline reaches 90 percent of its cap, which protects heartbeats on a nearly full pipeline. The run queues and is retried.
  • Scheduler starvation: the Error Dashboard banner threshold is an oldest wait of 10 minutes.
  • Queue timeout: a capacity-queued run older than its pipeline’s queue_timeout_seconds is superseded - status rejected, error code queue.expired. The default is null (never), so a capacity-queued run otherwise waits indefinitely. The window is anchored on created_at (when the run was queued) and applies only to capacity-marked runs; never-dispatched and enqueue-failed runs keep their own timeouts.
  • Queue expiry classification: queue_expired is its own reason and is excluded from failure counts and thresholds. It is not a reviewer rejection (hitl_rejected) and does not page anyone.
  • Re-dispatch cadence: a queued run is re-checked for capacity by a reconciler that runs every 60 seconds.
Next: Execution Environments