Feature Spotlight: How concurrency controls keep expensive agents from colliding

Concurrency controls limit how many pipeline runs can execute at the same time, at both the pipeline level and the organisation level. They exist because two agents fighting over the same database connection pool or burning through the same sandbox quota is not faster than running them one at a time. It is just more expensive and less reliable. Modulo enforces concurrency limits before a run starts, so a run either gets a slot or waits for one to free up. No overprovisioning, no surprise failures.

In short

• What it does: max_concurrent_runs caps how many runs a single pipeline can execute at once. An org-level sandbox cap limits total concurrent work across all pipelines.
• When you would use it: pipelines that call expensive LLMs, provision sandboxes, or hit shared infrastructure like Postgres and Redis.
• When you would not: read-only or lightweight pipelines where the cost of a concurrent run is negligible.
• How overflow is handled: runs that exceed the cap queue instead of failing, and start when a slot opens.

The problem it solves

Say you run three pipelines that all call the same model backend and write to the same Postgres database. Each pipeline is configured to run without limits. On a quiet day, two of them start at the same time and the third follows a minute later. The database connection pool saturates. Two runs succeed. The third times out with a lock error and retries, burning another round of model tokens for no reason. The next day, the same thing happens but with five concurrent runs. Now two fail, and the retry storm pushes your model spend up by 30% overnight.

The root cause is not that any one pipeline is broken. It is that nobody set a ceiling on how many runs could execute at once. Without that ceiling, the system runs as many jobs as it can start, not as many as the infrastructure can sustain. The result is unpredictable cost spikes, connection timeouts, and a team that spends mornings triaging failures instead of shipping features.

How it works

Concurrency in Modulo is enforced at two layers. The product layer controls what users see. The infrastructure layer controls what the deployment can actually handle.

Per-pipeline concurrency via max_concurrent_runs

Every pipeline has a max_concurrent_runs field. The default is 1, meaning one run at a time. Set it higher when a pipeline needs to process work in parallel, or when you want to allow overlapping runs without waiting for the previous one to finish.

{
  "name": "summarize-docs",
  "max_concurrent_runs": 3
}

The executor enforces this limit via a database lock (SELECT FOR UPDATE on the pipeline row), so two runs cannot slip past the check at the same time. When a run starts but no slot is available, the system checks whether one will free up within lock_wait_timeout_seconds. If so, the run is deferred and starts as soon as a slot opens. If not, the run fails cleanly with error_code="lock_timeout" rather than consuming resources it cannot use.

Org-level concurrency

An organisation-wide sandbox cap limits the total number of concurrent sandboxes across all pipelines. This is the global ceiling. Even if every pipeline has max_concurrent_runs: 5, the org cap ensures the total never exceeds what the deployment can handle.

Ongoing triggers and concurrency

Cron triggers, webhook triggers, and polling triggers all respect both the pipeline-level and org-level limits. A cron job that fires every 15 minutes will not start a new run if the pipeline is already at capacity. The run queues and starts when a slot opens. This keeps scheduled work from colliding with manual runs or other triggered pipelines.

Queue behaviour

When a run queues, it waits for the next available slot. It does not fail. As soon as an active run completes, the next queued run starts automatically. This means you can set conservative concurrency limits without blocking work entirely. The queue acts as a buffer, smoothing out bursts without overloading the system.

Journey example: before and after

Before

Jordan runs a data team at a mid-size SaaS company. The team has four pipelines: one that ingests customer feedback, one that summarizes support tickets, one that generates weekly reports, and one that syncs data to the warehouse. All four run without concurrency limits. On Monday mornings, the feedback ingestion and the weekly report both start at the same time. They fight for Postgres connections. The report pipeline retries three times before it succeeds, each retry burning model tokens. By Tuesday, Jordan's team has learned to manually stagger the runs by kicking them off 30 minutes apart. It works, but it means someone has to remember to start the pipelines in the right order, and it breaks the moment someone forgets.

After

Jordan sets max_concurrent_runs: 2 on the report pipeline and max_concurrent_runs: 3 on the feedback ingestion pipeline. The org sandbox cap is set to 6. The weekly report now queues behind the feedback ingestion instead of colliding with it. No retries. No wasted tokens. The team stops manually staggering runs and trusts the system to handle the queue. Jordan checks the analytics dashboard once a week to confirm the queue is draining as expected, and adjusts the limits when the team adds a new pipeline.

When to use it vs. when not to

Use concurrency limits on any pipeline that calls expensive resources: model backends with token-based pricing, sandboxes with per-use billing, or shared databases with connection pools. Use them on pipelines triggered by cron or webhooks, where a burst of triggers can pile up faster than the infrastructure can handle them.

Do not over-constrain lightweight pipelines. A pipeline that reads a file and returns a summary does not need a concurrency limit if it completes in under a second and touches no shared resources. Over-constraining creates unnecessary queuing and slows down work that could safely run in parallel. The goal is to match concurrency limits to the actual cost and shared-resource usage of each pipeline, not to set a blanket limit across the board.

Where it fits in a pipeline

Concurrency controls sit alongside governance gates and the audit trail. Where HITL gates decide whether a step should proceed, concurrency controls decide whether the infrastructure can support it. Together they form the operating contract between what the pipeline wants to do and what the deployment can actually sustain. For the broader context, see the principles of Modulo.

Go deeper

Read the concurrency docs for sizing formulas and infrastructure knobs, or the related docs on pipelines. Try the hosted demo at demo.modulo.run, or use the contact page to talk to us about tuning concurrency for your deployment.