Provider Cache Server

Terragrunt has the ability to cache OpenTofu/Terraform providers across all OpenTofu/Terraform instances, ensuring that each provider is only ever downloaded and stored on disk exactly once by running a local provider cache server while running Terragrunt.

Why caching is useful

Let’s imagine that your project consists of 50 Terragrunt modules (terragrunt.hcl), each of them uses the same provider aws. Without caching, each of them will download the provider from the Internet and stored in its own .terraform directory. For clarity, the downloadable archive terraform-provider-aws_5.36.0_darwin_arm64.zip has a size of ~100MB, and when unzipped it takes up ~450MB of disk space. It’s easy to calculate that initializing such a project with 50 modules will cost you 5GB of traffic and 22.5GB of free space instead of 100MB and 450MB using the cache.

Why OpenTofu/Terraform’s built-in provider caching doesn’t work

OpenTofu/Terraform has provider caching feature Provider Plugin Cache, that does the job well until you run multiple OpenTofu/Terraform processes simultaneously, such as when you use terragrunt run-all. Then the OpenTofu/Terraform processes begin conflict by overwriting each other’s cache, which causes the error like Error: Failed to install provider. As a result, Terragrunt previously had to disable concurrency for init steps in run-all, which is significantly slower. If you enable Terragrunt Provider Caching, as described in this section, that will no longer be necessary, and you should see significant performance improvements with init, as well as significant savings in terms of bandwidth and disk space usage.

Usage

Terragrunt Provider Cache is currently considered an experimental feature, so it is disabled by default. To enable it you need to use the flag terragrunt-provider-cache:

terragrunt run-all apply --terragrunt-provider-cache

or the environment variable TERRAGRUNT_PROVIDER_CACHE:

TERRAGRUNT_PROVIDER_CACHE=1 terragrunt run-all apply

By default, cached providers are stored in terragrunt/providers folder, which is located in the user cache directory:

• $HOME/.cache/terragrunt/providers on Unix systems
• $HOME/Library/Caches/terragrunt/providers on Darwin
• %LocalAppData%\terragrunt\providers on Windows

The file structure of the cache directory is identical to the OpenTofu/Terraform plugin_cache_dir directory. If you already have a directory with providers cached by OpenTofu/Terraform plugin_cache_dir, you can set this path using the flag terragrunt-provider-cache-dir, to make cache server reuse them.

terragrunt plan \
--terragrunt-provider-cache \
--terragrunt-provider-cache-dir /new/path/to/cache/dir

or the environment variable TERRAGRUNT_PROVIDER_CACHE_DIR:

TERRAGRUNT_PROVIDER_CACHE=1 \
TERRAGRUNT_PROVIDER_CACHE_DIR=/new/path/to/cache/dir \
terragrunt plan

By default, Terragrunt only caches providers from the following registries: registry.terraform.io, registry.opentofu.org. You can override this list using the flag terragrunt-provider-cache-registry-names:

terragrunt apply \
--terragrunt-provider-cache \
--terragrunt-provider-cache-registry-names example1.com \
--terragrunt-provider-cache-registry-names example2.com

or the environment variable TERRAGRUNT_PROVIDER_CACHE_REGISTRY_NAMES:

TERRAGRUNT_PROVIDER_CACHE=1 \
TERRAGRUNT_PROVIDER_CACHE_REGISTRY_NAMES=example1.com,example2.com \
terragrunt apply

How Terragrunt Provider Caching works

• Start a server on localhost. This is the Terragrunt Provider Cache server.

• Configure OpenTofu/Terraform instances to use the Terragrunt Provider Cache server as a remote registry:

  • Create local CLI config file .terraformrc for each module that concatenates the user configuration from the OpenTofu/Terraform CLI config file with additional sections:

  • provider-installation forces OpenTofu/Terraform to look for for the required providers in the cache directory and create symbolic links to them, if not found, then request them from the remote registry.
  • host forces OpenTofu/Terraform to forward all provider requests through the Terragrunt Provider Cache server. The address link contains UUID and is unique for each module, used by Terragrunt Provider Cache server to associate modules with the requested providers.
  • Set environment variables:
    • TF_CLI_CONFIG_FILE sets to use just created local CLI config .terragrunt-cache/.terraformrc
    • TFTOKEN* sets per-remote-registry tokens for authentication to Terragrunt Provider Cache server.
• Any time Terragrunt is going to run init:
  • Call terraform init. This gets OpenTofu/Terraform to request all the providers it needs from the Terragrunt Provider Cache server.
  • The Terragrunt Provider Cache server will download the provider from the remote registry, unpack and store it into the cache directory or create a symlink if the required provider exists in the user plugins directory. Note that the Terragrunt Provider Cache server will ensure that each unique provider is only ever downloaded and stored on disk once, handling concurrency (from multiple OpenTofu/Terraform and Terragrunt instances) correctly. Along with the provider, the cache server downloads hashes and signatures of the providers to check that the files are not corrupted.
  • The Terragrunt Provider Cache server returns the HTTP status 429 Locked to OpenTofu/Terraform. This is because we do not want OpenTofu/Terraform to actually download any providers as a result of calling terraform init; we only use that command to request the Terragrunt Provider Cache Server to start caching providers.
  • At this point, all providers are downloaded and cached, so finally, we run terragrunt init a second time, which will find all the providers it needs in the cache, and it’ll create symlinks to them nearly instantly, with no additional downloading.
  • Note that if a OpenTofu/Terraform module doesn’t have a lock file, OpenTofu/Terraform does not use the cache, so it would end up downloading all the providers from scratch. To work around this, we generate .terraform.lock.hcl based on the request made by terrafrom init to the Terragrunt Provider Cache server. Since terraform init only requests the providers that need to be added/updated, we can keep track of them using the Terragrunt Provider Cache server and update the OpenTofu/Terraform lock file with the appropriate hashes without having to parse tf configs.

Reusing providers from the user plugins directory

Some plugins for some operating systems may not be available in the remote registries. Thus, the cache server will not be able to download the requested provider. As an example, plugin template v2.2.0 for darwin-arm64, see Template v2.2.0 does not have a package available - Mac M1. The workaround is to compile the plugin from source code and put it into the user plugins directory or use the automated solution https://github.com/kreuzwerker/m1-terraform-provider-helper. For this reason, the cache server first tries to create a symlink from the user’s plugin directory if the required provider already exists there:

• %APPDATA%\terraform.d\plugins on Windows
• ~/.terraform.d/plugins on other systems

How forwarding OpenTofu/Terraform requests through the Terragrunt Provider Cache works

OpenTofu/Terraform has an official documented setting network_mirror, that works great, but has one major drawback for the local cache server - the need to use https connection with a trusted certificate. Fortunately, there is another way - using the undocumented host setting, which allows OpenTofu/Terraform to create connections to the caching server over HTTP.

Provider Cache with providers lock command

If you run providers lock with enabled Terragrunt Provider Cache, Terragrunt creates the provider cache and generates the lock file on its own, without running terraform providers lock at all.

terragrunt providers lock -platform=linux_amd64 -platform=darwin_arm64 -platform=freebsd_amd64 \
--terragrunt-provider-cache

Configure the Terragrunt Cache Provider

Since Terragrunt Provider Cache is essentially a Private Registry server that accepts requests from OpenTofu/Terraform, downloads and saves providers to the cache directory, there are a few more flags that are unlikely to be needed, but are useful to know about:

• Server Hostname terragrunt-provider-cache-hostname, by default, localhost.
• Server Port terragrunt-provider-cache-port, assigned automatically every time you launch the Terragurnt.
• Server Token terragrunt-provider-cache-token, generated automatically every time you launch the Terragurnt.

To enhance security, the Terragrunt Provider Cache has authentication to prevent unauthorized connections from third-party applications. You can set your own token using any character set.

terragrunt apply \
--terragrunt-provider-cache \
--terragrunt-provider-cache-host 192.168.0.100 \
--terragrunt-provider-cache-port 5758 \
--terragrunt-provider-cache-token my-secret

or using environment variables:

TERRAGRUNT_PROVIDER_CACHE=1 \
TERRAGRUNT_PROVIDER_CACHE_HOST=192.168.0.100 \
TERRAGRUNT_PROVIDER_CACHE_PORT=5758 \
TERRAGRUNT_PROVIDER_CACHE_TOKEN=my-secret \
terragrunt apply