How to test a Terraform Configuration?

If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / Continuous Delivery pipeline?

As an example, you may have a multi-cloud configuration that specifies the following desired state:

• Azure Container Services to host Docker in Azure

• Azure Blob Storage

• SQL Azure

• EC2 Container Service to host Docker in AWS

• Amazon S3 Storage Service

• Amazon RDS SQL Server database

Potentially terraform apply could create the above from scratch, or transition from a partially deployed state to the above desired state.

I am aware that Terraform splits its work into the execution plan stage and the application phase which actually makes changes to the target architecture. Can this be used to write tests against the execution plan, if so are there frameworks to help write these?

There is currently no full solution to this integrated into Terraform, but there are some building blocks that could be useful to assist in writing tests in a separate programming language.

Terraform produces state files in JSON format that can, in principle, be used by external programs to extract certain data about what Terraform created. While this format is not yet considered officially stable, in practice it changes infrequently enough that people have successfully integrated with it, accepting that they might need to make adjustments as they upgrade Terraform.

What strategy is appropriate here will depend a lot on what exactly you want to test. For example:

-
In an environment that's spinning up virtual servers, tools like Serverspec can be used to run tests from the perspective of these servers. This can either be run separately from Terraform using some out-of-band process, or as part of the Terraform apply using the remote-exec provisioner. This allows verification of questions like "can the server reach the database?", but is not suitable for questions such as "is the instance's security group restrictive enough?", since robustly checking that requires accessing data from outside of the instance itself.

-
It's possible to write tests using an existing test framework (such as RSpec for Ruby, unittest for Python, etc) which gather relevant resource ids or addresses from the Terraform state file and then use the relevant platform's SDK to retrieve data about the resources and assert that they are set up as expected. This is a more general form of the previous idea, running the tests from the perspective of a host outside of the infrastructure under test, and can thus collect a broader set of data to make assertions on.

-
For more modest needs, one can choose to trust that the Terraform state is an accurate representation of reality (a valid assumption in many cases) and simply assert directly on that. This is most appropriate for simple "lint-like" cases, such as verifying that the correct resource tagging scheme is being followed for cost-allocation purposes.

There is some more discussion about this in a relevant Terraform Github issue.

In the latest versions of Terraform it is strongly recommended to use a remote backend for any non-toy application, but that means that the state data is not directly available on local disk. However, a snapshot of it can be retrieved from the remote backend using the terraform state pull command, which prints the JSON-formatted state data to stdout so it can be captured and parsed by a calling program.

StackExchange DevOps Q#863, answer score: 24