Shape
Shape

Keep your Terragrunt Architecture DRY

Dry Use cases Backend

Keep your Terragrunt Architecture DRY

Motivation

As covered in Keep your Terraform code DRY and Keep your remote state configuration DRY, it becomes important to define base Terragrunt configuration files that are included in the child config. For example, you might have a root Terragrunt configuration that defines the remote state and provider configurations:

remote_state {
  backend = "s3"
  config = {
    bucket         = "my-terraform-state"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  assume_role {
    role_arn = "arn:aws:iam::0123456789:role/terragrunt"
  }
}
EOF
}

You can then include this in each of your child terragrunt.hcl files using the include block for each infrastructure module you need to deploy:

include "root" {
  path = find_in_parent_folders()
}

This pattern is useful for global configuration blocks that need to be included in all of your modules, but what if you have Terragrunt configurations that are only relevant to subsets of your module? For example, consider the following terragrunt file structure, which defines three environments (prod, qa, and stage) with the same infrastructure in each one (an app, a MySQL database, and a VPC):

└── live
    ├── terragrunt.hcl
    ├── prod
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    ├── qa
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    └── stage
        ├── app
        │   └── terragrunt.hcl
        ├── mysql
        │   └── terragrunt.hcl
        └── vpc
            └── terragrunt.hcl

More often than not, each of the services will look similar across the different environments, only requiring small tweaks. For example, the app/terragrunt.hcl files may be identical across all three environments except for an adjustment to the instance_type parameter for each environment. These identical settings don’t belong in the root terragrunt.hcl configuration because they are only relevant to the app configurations, and not mysql or vpc. However, it is cumbersome to copy paste these settings across all three environments.

To solve this, you can use multiple include blocks.

Using include to DRY common Terragrunt config

Suppose your qa/app/terragrunt.hcl configuration looks like the following:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "github.com/<org>/modules.git//app?ref=v0.1.0"
}

dependency "vpc" {
  config_path = "../vpc"
}

dependency "mysql" {
  config_path = "../mysql"
}

inputs = {
  env            = "qa"
  basename       = "example-app"
  vpc_id         = dependency.vpc.outputs.vpc_id
  subnet_ids     = dependency.vpc.outputs.subnet_ids
  mysql_endpoint = dependency.mysql.outputs.endpoint
}

In this example, the only thing that is different between the environments is the env input variable. This means that except for one line, everything in the config is duplicated across prod, qa, and stage.

To DRY this up, we will introduce a new folder called _env which will contain the common configurations across the three environments (we prefix with _ to indicate that this folder doesn’t contain deployable configurations):

└── live
    ├── terragrunt.hcl
    ├── _env
    │   ├── app.hcl
    │   ├── mysql.hcl
    │   └── vpc.hcl
    ├── prod
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    ├── qa
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    └── stage
        ├── app
        │   └── terragrunt.hcl
        ├── mysql
        │   └── terragrunt.hcl
        └── vpc
            └── terragrunt.hcl

In our example, the contents of _env/app.hcl would look like the following:

terraform {
  source = "github.com/<org>/modules.git//app?ref=v0.1.0"
}

dependency "vpc" {
  config_path = "../vpc"
}

dependency "mysql" {
  config_path = "../mysql"
}

inputs = {
  basename       = "example-app"
  vpc_id         = dependency.vpc.outputs.vpc_id
  subnet_ids     = dependency.vpc.outputs.subnet_ids
  mysql_endpoint = dependency.mysql.outputs.endpoint
}

Note that everything is defined except for the env input variable. We now modify qa/app/terragrunt.hcl to include this alongside the root configuration by using multiple include blocks, significantly reducing our per environment configuration:

include "root" {
  path = find_in_parent_folders()
}

include "env" {
  path = "${get_terragrunt_dir()}/../../_env/app.hcl"
}

inputs = {
  env = "qa"
}

Using exposed includes to override common configurations

In the previous section, we covered using include to DRY common component configurations. While powerful, include has a limitation where the included configuration is statically merged into the child configuration.

In our example, note that the _env/app.hcl file hardcodes the app module version to v0.1.0 (relevant section pasted below for convenience):

terraform {
  source = "github.com/<org>/modules.git//app?ref=v0.1.0"
}

# ... other blocks omitted for brevity ...

What if we want to deploy a different version for each environment? One way you can do this is by redefining the terraform block in the child config. For example, if you want to deploy v0.2.0 in the qa environment, you can do the following:

include "root" {
  path = find_in_parent_folders()
}

include "env" {
  path = "${get_terragrunt_dir()}/../../_env/app.hcl"
}

# Override the terraform.source attribute to v0.2.0
terraform {
  source = "github.com/<org>/modules.git//app?ref=v0.2.0"
}

inputs = {
  env = "qa"
}

While this works, we now have duplicated the source URL. To avoid repeating the source URL, we can use exposed includes to reference data defined in the parent configurations. To do this, we will refactor our parent configuration to expose the source URL as a local variable instead of defining it into the terraform block:

locals {
  source_base_url = "github.com/<org>/modules.git//app"
}

# ... other blocks and attributes omitted for brevity ...

We then set the expose attribute to true on the include block in the child configuration so that we can reference the defined data in the parent configuration. Using that, we can construct the terraform source URL without having to repeat the module source:

include "root" {
  path = find_in_parent_folders()
}

include "env" {
  path   = "${get_terragrunt_dir()}/../../_env/app.hcl"
  expose = true
}

# Construct the terraform.source attribute using the source_base_url and custom version v0.2.0
terraform {
  source = "${include.env.locals.source_base_url}?ref=v0.2.0"
}

inputs = {
  env = "qa"
}

Using read_terragrunt_config to DRY parent configurations

In the previous two sections, we covered using include to DRY common component configurations through static merges with the child configuration. What if you want to dynamically update the parent configuration without having to define the override blocks in the child config?

In our example, the child configuration defines the env input in its configuration (pasted below for convenience):

# ... other blocks omitted for brevity ...

inputs = {
  env = "qa"
}

What if some inputs depend on this env input? For example, what if we want to append the env to the name input prior to passing to terraform? One way is to define the override parameters in the child config instead of the parent:

# ... other blocks omitted for brevity ...

include "env" {
  path   = "${get_terragrunt_dir()}/../../_env/app.hcl"
  expose = true
}

inputs = {
  env      = "qa"
  basename = "${include.env.locals.basename}-qa"
}

While this works, you could lose all the DRY advantages of the include block if you have many configurations that depend on the env input. Instead, you can use read_terragrunt_config to load additional context into the the parent configuration by taking advantage of the folder structure, and define the env based logic in the parent configuration.

To do this, we will introduce a new env.hcl configuration in each environment:

└── live
    ├── terragrunt.hcl
    ├── _env
    │   ├── app.hcl
    │   ├── mysql.hcl
    │   └── vpc.hcl
    ├── prod
    │   ├── env.hcl
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    ├── qa
    │   ├── env.hcl
    │   ├── app
    │   │   └── terragrunt.hcl
    │   ├── mysql
    │   │   └── terragrunt.hcl
    │   └── vpc
    │       └── terragrunt.hcl
    └── stage
        ├── env.hcl
        ├── app
        │   └── terragrunt.hcl
        ├── mysql
        │   └── terragrunt.hcl
        └── vpc
            └── terragrunt.hcl

The env.hcl configuration will look like the following:

locals {
  env = "qa" # this will be prod in the prod folder, and stage in the stage folder.
}

We can then load the env.hcl file in the _env/app.hcl file to load the env string:

locals {
  # Load the relevant env.hcl file based on where terragrunt was invoked. This works because find_in_parent_folders
  # always works at the context of the child configuration.
  env_vars = read_terragrunt_config(find_in_parent_folders("env.hcl"))
  env_name = local.env_vars.locals.env

  source_base_url = "github.com/<org>/modules.git//app"
}

dependency "vpc" {
  config_path = "../vpc"
}

dependency "mysql" {
  config_path = "../mysql"
}

inputs = {
  env            = local.env_name
  basename       = "example-app-${local.env_name}"
  vpc_id         = dependency.vpc.outputs.vpc_id
  subnet_ids     = dependency.vpc.outputs.subnet_ids
  mysql_endpoint = dependency.mysql.outputs.endpoint
}

With this configuration, env_vars is loaded based on which folder is being invoked. For example, when Terragrunt is invoked in the prod/app/terragrunt.hcl folder, prod/env.hcl is loaded, while qa/env.hcl is loaded when Terragrunt is invoked in the qa/app/terragrunt.hcl folder.

Now we can clean up the child config to eliminate the env input variable since that is loaded in the env.hcl context:

include "root" {
  path = find_in_parent_folders()
}

include "env" {
  path   = "${get_terragrunt_dir()}/../../_env/app.hcl"
  expose = true
}

# Construct the terraform.source attribute using the source_base_url and custom version v0.2.0
terraform {
  source = "${include.env.locals.source_base_url}?ref=v0.2.0"
}

Considerations for CI/CD Pipelines

For infrastructure CI/CD pipelines, it is common to only want to run the workflow on the modules that were updated. For example, if you only changed the terragrunt.hcl configuration for the RDS database in the dev account, then you only want to run plan and apply on that module, not other components or other accounts.

If you did not take advantage of include or read_terragrunt_config, then implementing this pipeline is straightforward: you can use git diff to collect all the files that changed, and for those terragrunt.hcl files that were updated, you can run terragrunt plan or terragrunt apply by passing in the updated file with --terragrunt-config.

However, if you use include or read_terragrunt_config, then a single file change may need to be reflected on multiple files that were not touched at all in the commit. In our previous example, when a configuration is updated in the _env/app.hcl file, we need to apply the change to all the modules that include that common environment configuration.

Terragrunt currently does not have any features for supporting this use case when read_terragrunt_config is used. However, for include blocks, you can use the –terragrunt-modules-that-include CLI option for the run-all command.

In the previous example, your CI/CD pipeline can run terragrunt run-all plan --terragrunt-modules-that-include _env/app.hcl. This will:

  • Recursively find all Terragrunt modules in the current directory tree.
  • Filter out any modules that don’t include _env/app.hcl so that they won’t be touched.
  • Run plan on any modules remaining (which will be the set of modules in the current tree that include _env/app.hcl).

Thereby allowing you to only touch those modules that need to be updated by the code change.

Alternatively, you can implement a promotion workflow if you have multiple environments that depend on the _env/app.hcl configuration. In the above example, suppose you wanted to progressively roll out the changes through the environments, qa, stage, and prod in order. In this case, you can use --terragrunt-working-dir to scope down the updates from the common file:

# Roll out the change to the qa environment first
terragrunt run-all plan --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir qa
terragrunt run-all apply --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir qa
# If the apply succeeds to qa, move on to the stage environment
terragrunt run-all plan --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir stage
terragrunt run-all apply --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir stage
# And finally, prod.
terragrunt run-all plan --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir prod
terragrunt run-all apply --terragrunt-modules-that-include _env/app.hcl --terragrunt-working-dir prod

This allows you to have flexibility in how changes are rolled out. For example, you can add extra validation stages inbetween the roll out to each environment, or add in manual approval between the stages.

NOTE: If you identify an issue with rolling out the change in a downstream environment, and want to abort, you will need to make sure that that environment uses the older version of the common configuration. This is because the common configuration is now partially rolled out, where some environments need to use the new updated common configuration, while other environments need the old one. The best way to handle this situation is to create a new copy of the common configuration at the old version and have the environments that depend on the older version point to that version.