Skip to main content

Advanced concurrency configuration

Use job metadata and schedules to prevent runs from starting if another run is already occurring (advanced)​

You can use Dagster's rich metadata to use a schedule or a sensor to only start a run when there are no currently running jobs.

src/<project_name>/defs/assets.py
import time

import dagster as dg


@dg.asset
def first_asset(context: dg.AssetExecutionContext):
    # sleep so that the asset takes some time to execute
    time.sleep(75)
    context.log.info("First asset executing")


my_job = dg.define_asset_job("my_job", [first_asset])


@dg.schedule(
    job=my_job,
    # Runs every minute to show the effect of the concurrency limit
    cron_schedule="* * * * *",
)
def my_schedule(context):
    # Find runs of the same job that are currently running
    run_records = context.instance.get_run_records(
        dg.RunsFilter(
            job_name="my_job",
            statuses=[
                dg.DagsterRunStatus.QUEUED,
                dg.DagsterRunStatus.NOT_STARTED,
                dg.DagsterRunStatus.STARTING,
                dg.DagsterRunStatus.STARTED,
            ],
        )
    )
    # skip a schedule run if another run of the same job is already running
    if len(run_records) > 0:
        return dg.SkipReason(
            "Skipping this run because another run of the same job is already running"
        )
    return dg.RunRequest()

Limit concurrent runs in branch deployments (Dagster+ only)​

In Dagster+, branch deployments share a set of organization-scoped concurrency settings that apply to every branch deployment in the organization:

  • max_concurrent_branch_deployment_runs: caps the total number of concurrent runs across all branch deployments (default 50).
  • max_concurrent_runs_per_branch_deployment: caps the number of concurrent runs within an individual branch deployment (default no limit).
  • branch_deployment_tag_concurrency_limits: caps the number of concurrent runs with matching run tags across all branch deployments.

These settings are useful for preventing branch deployments from consuming too many resources, especially when multiple developers are working simultaneously.

For details on each setting and how to edit them in the Dagster+ UI or with the dg CLI, see Configuring concurrency for branch deployments.