Run Queue

Terragrunt’s “Run Queue” is the mechanism it uses to manage the run order and concurrency when running OpenTofu/Terraform commands across multiple Terragrunt units. This is particularly relevant when using the run --all or run --graph commands.

How it Works: The Dependency Graph (DAG)

At its core, the Run Queue relies on a Directed Acyclic Graph (DAG) built from the dependencies defined between your Terragrunt units. These dependencies are typically established using dependency or dependencies blocks in your terragrunt.hcl files.

Terragrunt analyzes these dependencies to determine the correct order of operations:

1. Discovery: Terragrunt discovers configurations that might be relevant to a run based on the current working directory.
2. Constructing the Queue: Based on the command being run, Terragrunt creates an ordered queue.
   • For commands like plan or apply, dependencies are run before the units that depend on them.
   • For commands like destroy, dependent units are run before their dependencies.
3. Runs: Terragrunt dequeues the units in the queue and runs them, respecting the queue order. By default, it runs units concurrently up to a certain limit (controlled by the --parallelism flag), but it will always wait for a unit’s dependencies (or dependents for destroys) to complete successfully before running that unit.

Example DAG

Consider a setup where:

• Unit “dependent” depends on unit “dependency”.
• Unit “dependency” depends on unit “ancestor-dependency”.
• Unit “independent” has no dependencies nor dependents.

• Directoryroot

  • Directorysubtree

    • Directorydependent

      • terragrunt.hcl
    • Directorydependency

      • terragrunt.hcl
  • Directoryancestor-dependency

    • terragrunt.hcl
  • Directoryindependent

    • terragrunt.hcl

Assuming a current working directory of the root directory, Terragrunt would run units in the following order:

• run --all plan Order: Terragrunt would run independent and ancestor-dependency concurrently. Once ancestor-dependency finishes, dependency would run. Once dependency finishes, dependent would run.
• run --all destroy Order: Terragrunt would run dependent and independent concurrently. Once dependent finishes, dependency would run. Once dependency finishes, ancestor-dependency would run.

Controlling the Queue

Several flags allow you to customize how Terragrunt builds and executes the run queue. By default, Terragrunt will include all units that are in the current working directory.

Include by default

By default, when using the --all flag, Terragrunt will include all units that are in the current working directory.

Using any positive filter triggers “Exclude by default” behavior, meaning that Terragrunt will no longer automatically include all units in the current working directory, and will instead selectively include units only if they match a positive filter (and don’t match any negative filters).

More on this in the Filtering Units section.

Filtering Units

You can control which units are included or excluded from the queue using the --filter flag.

Positive Filters

Any filter that doesn’t start with a ! prefix is considered a positive filter. When Terragrunt sees that any positive filter is present, it will evaluate every path it encounters while walking the directory tree, and only include units that match a positive filter.

terragrunt run --all --filter './subtree/**' -- plan

This will tell Terragrunt only to run the units that match the glob pattern ./subtree/** (any directory found in the current working directory starting with subtree).

There are many more types of filters you can use, and those details are covered in the Filter feature documentation.

Negative Filters

Any filter that starts with a ! prefix is considered a negative filter. Negative filters are always evaluated after positive filters (if present).

terragrunt run --all --filter '!./subtree/**' -- plan

This will tell Terragrunt to run all units in the current working directory except those that match the glob pattern ./subtree/** (any directory found in the current working directory starting with subtree).

Combining Positive and Negative Filters

You can combine positive and negative filters to ensure that only the units you want are run.

terragrunt run --all --filter './subtree/**' --filter '!./subtree/dependency/**' -- plan

This will tell Terragrunt to run all units in the directory subtree except those that match the glob pattern ./subtree/dependency/** (any directory found in the current working directory starting with subtree/dependency).

Modifying Order and Error Handling

• --queue-construct-as (--as): Build the run queue as if a particular command was run. Useful for performing dry-runs of run using discovery commands, like find and list.

  e.g. terragrunt list --queue-construct-as destroy

  This lists the units in the order they’d be processed for run --all destroy:

  $ terragrunt list --as destroy -l
  Type  Path
  unit  independent
  unit  subtree/dependent
  unit  subtree/dependency
  unit  ancestor-dependency

  $ terragrunt list --as plan -l
  Type  Path
  unit  ancestor-dependency
  unit  independent
  unit  subtree/dependency
  unit  subtree/dependent

• --queue-ignore-dag-order: Run units in the queue concurrently without respecting the dependency order.

  e.g. terragrunt run --all plan --queue-ignore-dag-order

  Run plan on ancestor-dependency, subtree/dependency, subtree/dependent, and independent all concurrently, without waiting for their defined dependencies. For instance, subtree/dependent’s plan would not wait for subtree/dependency’s plan to complete.

  Caution

  This flag is useful for faster runs in stateless commands like validate or plan, but is dangerous for commands that modify state like apply or destroy.

  You might encounter failed applies if unit dependencies are not applied before dependents, and conversely, failed destroys if unit dependents are not destroyed before dependencies.

• --queue-ignore-errors: Continue processing the queue even if some units fail.

  e.g. terragrunt run --all plan --queue-ignore-errors

  If ancestor-dependency’s plan fails, Terragrunt will still attempt to run plan for subtree/dependency, then subtree/dependent, and also for independent.

  Caution

  This flag is useful for identifying all errors at once, but can lead to inconsistent state if used with apply or destroy.

  You might encounter failed applies if unit dependencies are not applied successfully before dependents, and conversely, failed destroys if unit dependents are not destroyed successfully before dependencies.

• --fail-fast: Fail the run if any unit fails, stopping all remaining units immediately.

  e.g. terragrunt run --all plan --fail-fast

  If independent’s plan fails, Terragrunt will stop running any more units and fail the run, even if ancestor-dependency’s plan succeeds.

  Tip

  This flag is useful when you want to fail the run as soon as any unit fails, and stop running any more units.

Important Considerations

Caution

When using run --all plan with units that have dependencies (e.g. via dependency or dependencies blocks), the command will fail if those dependencies have never been deployed. This is because Terragrunt cannot resolve dependency outputs without existing state.

To work around this issue, use mock outputs in dependency blocks.

Caution

Do not set TF_PLUGIN_CACHE_DIR when using run --all (unless using OpenTofu >= 1.10).

This can cause concurrent access issues with the provider cache. Instead, use Terragrunt’s built-in Provider Cache Server.

Caution

When using run --all with apply or destroy, Terragrunt automatically adds the -auto-approve flag due to limitations with shared stdin making individual approvals impossible. Use --no-auto-approve to override this, but be aware you might need alternative approval workflows.