Terraform media - how to do it DRY

Question

We use Terraform to a large extent to provide AWS Cloud. Our basic structure for terraforms is as follows:

├─ modules ├── x ├── y ├─ environments ├── dev │ ├── main.tf │ ├── output.tf │ └── variables.tf └── uat │ ├── main.tf │ ├── output.tf │ └── variables.tf └── prod ├── main.tf ├── output.tf └── variables.tf 

As we reach a point where we have many modules and many environments, code duplication is now becoming a more serious headache, we would like to get rid of as many as possible.

Our main problem is currently related to output.tf files - every time we expand an existing module or add a new module, we need to configure a specific configuration for it (this is expected), but we still have to copy / paste the necessary parts into output.tf to display initialization results (e.g. IP addresses, ARS ARS, etc.).

Is there a way to get rid of duplicate output.tf files? Can we just define the desired outputs in the modules themselves and see all the specific outputs whenever we run terraform for a particular environment?

Answer 1

One way to solve this problem is to create a base environment and then symbolize common elements, for example:

 ├─ modules ├── x ├── y ├─ environments ├── base │ ├── output.tf │ └── variables.tf ├── dev │ ├── main.tf │ ├── output.tf -> ../base/output.tf │ └── variables.tf -> ../base/variables.tf ├── uat │ ├── main.tf │ ├── output.tf -> ../base/output.tf │ └── variables.tf -> ../base/variables.tf ├── super_custom │ ├── main.tf │ ├── output.tf # not symlinked │ └── variables.tf # not symlinked └── prod ├── main.tf ├── output.tf -> ../base/output.tf └── variables.tf -> ../base/variables.tf 

This approach really only works if your output.tf and variables.tf files output.tf same for each environment, and although you may have asymmetric options (like super_custom above), this can be confusing as it is not immediately obvious which environments are common. and which are not. YMMV. I am trying to save changes between environments limited by the .tfvars file for each environment.

It’s worth reading Charity Major's excellent post in the tfstate files that set me this way.

Answer 2

If your dev , uat and prod environments have the same shape but different properties, you can use workspaces to separate your environment state, as well as separate *.tfvars files to specify different configurations.

It might look like this:

 ├─ modules │ ├── x │ └── y ├── dev.tfvars ├── prod.tfvars ├── uat.tfvars ├── main.tf ├── outputs.tf └── variables.tf 

You can create a new workspace with:

 terraform workspace new uat 

Then the deployment of changes will be:

 terraform workspace select uat terraform apply --var-file=uat.tfvars 

The function of workspaces ensures that different environmental conditions are managed separately, which is a bonus.

This approach only works when the differences between the environments are small enough, which makes sense to encapsulate the logic for this in separate modules (for example, having the high_availability flag, which adds some additional redundant infrastructure for uat and prod ).

Answer 3

We created and opened the source Terragrunt to solve this problem. One of the features of Terragrunt is the ability to load remote Terraform configurations. The idea is that you only define Terraform code for your infrastructure once, in one repo, called, for example, modules :

 └── modules ├── app │ └── main.tf ├── mysql │ └── main.tf └── vpc └── main.tf 

This repo contains typical Terraform code with one difference: everything in your code, which should be different between environments, should display as an input variable. For example, an application module may display the following variables:

 variable "instance_count" { description = "How many servers to run" } variable "instance_type" { description = "What kind of servers to run (eg t2.large)" } 

In a separate repo, called, for example, live, you define code for all your environments, which now consists of just one .tfvars file for each component (for example, app/terraform.tfvars , mysql/terraform.tfvars , etc.) . This gives you the following file layout:

 └── live ├── prod │ ├── app │ │ └── terraform.tfvars │ ├── mysql │ │ └── terraform.tfvars │ └── vpc │ └── terraform.tfvars ├── qa │ ├── app │ │ └── terraform.tfvars │ ├── mysql │ │ └── terraform.tfvars │ └── vpc │ └── terraform.tfvars └── stage ├── app │ └── terraform.tfvars ├── mysql │ └── terraform.tfvars └── vpc └── terraform.tfvars 

Please note that in any of the folders there are no Terraform configurations ( .tf files). Instead, each .tfvars file specifies a terraform { ... } block terraform { ... } , which indicates where to download the Terraform code, as well as the environment values ​​for the input variables of this Terraform code. For example, stage/app/terraform.tfvars might look like this:

 terragrunt = { terraform { source = "git:: [e-mail] :foo/modules.git//app?ref=v0.0.3" } } instance_count = 3 instance_type = "t2.micro" 

And prod/app/terraform.tfvars might look like this:

 terragrunt = { terraform { source = "git:: [e-mail] :foo/modules.git//app?ref=v0.0.1" } } instance_count = 10 instance_type = "m2.large" 

See the Terragrunt documentation for more information.