Infrastructure as code and TDD

Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while resolving the differences.

In Salt Stack those bits are called states. If state does not match the reality, the tool will resolve it for us. In other words - we are writing a test for our infrastructure and if the test fails, the tool will fix it on its own. At least that's the idea.

XP teaches us to use TDD and the question is if it's applicable to infrastructure? The tooling suggests it is.

I can imagine few type of tests that can be very useful.

We write smoke tests that are bundled with the deployed service to ensure that end-to-end the deployed service works and runs as expected. This would be an API call or/and systemctl check to make sure what we just deployed works. A lot of this functionality can be covered in the same states since tools like ansible have states to make sure a service is running.

There is project Molecule that allows running individual roles (as ansible calls its states) against docker or another temporary virtualisation engine. This forces to decouple roles and allows to execute them in isolation from the playbook while working on them. Tests mostly allow mocking the variables that the role is supposed to work with. Other examples seem like a duplication of the ansible engine though (assert a file belongs to a user...).

ThoughtWorks tech radar right now praises tools like inspec, serverspec or goss for validating that the server meets the spec. But we are writing a spec, aren't we?

So is there a point in further testing infrastructure if we are describing infrastructure in states/roles? I could suspect this becomes more required in larger organisations where one team provides the spec and other follows, or if there is a large set of roles maybe you want to run a subset of those and get a speed benefit from tests? I'm struggling to see why

In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.

First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.

Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended apt-get install side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the ps outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.

These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.

StackExchange DevOps Q#2777, answer score: 6