How to test provisioning and configuration in Ansible setup?

Looking at trying to build some resilience into our Ansible setup which deals with provisioning and configuration.

I understand a few methods of testing on the configuration side of things but I'm wondering how best to implement testing on the provisioning side of things, and if there are any tools which can help with this type of implementation.

Currently a lot of our testing is done serially during the playbook which makes a lot of sense for stuff like "has service come up; is the vip available; has this async task finished" but what really concerns me is our ability to manage drift of configuration at both the application and provisioning layer (such as VM configuration). I'm aware Ansible isn't the best tool for working with configuration drift but I'm curious to see your own opinions.

If you have something to fully automate the process even better. (we have a few ugly scripts which report back in slack daily).

Note: Right now we have a few conditions where a reprovision might occur (e.g. rebuild from backup, critical systems issue) but typically it just loops through some of the ansible configuring tasks and thinks no more of it.

Some options out there..

Testing tools: Sorted by github stars

• Serverspec - Ruby, most popular tool out there, built on ruby's rspec

• Goss - YAML, simple,

• Inspec - Ruby, think of it as an improved serverspec, almost same syntax, made by the chef guys. Built to be easier to extend than serverspec

• Testinfra - Python, has the cool feature of being able to use Ansible's inventory/vars

Major differences between them:

Ultimately, I would suggest spending a day experimenting with all of them to get a feel for them before deciding for yourself.

• With the exception of Goss, all the frameworks can run against a remote machine (ex. over ssh). Goss only runs locally or in docker w/ dgoss.

• All frameworks can be run locally on a server, but require Python or Ruby to be installed or embedded. Inspec provides a self-contained

• Goss has built in support for nagios/sensu output, this allows for easier integration with monitoring tools.

• Goss tests tend to be simpler, but less flexible since it's based on YAML. Other frameworks allow you to leverage the full power of the underlying language Python/Ruby to write tests or extend the tool's functionality. (simplicity vs flexibility)

• Goss allows you to generate tests from current system state

• Testinfra to my knowledge is the only one that has built-in support for ansible inventory and variables

• Inspec is backed by Chef

Continuous/divergence testing:

• Chef Compliance - works with inspec to continuously test your servers, paid product

• Goss - Can be easily hooked into Nagios or Sensu. Also, supports exposing server tests as an http endpoint.

Testing harnesses for development:

• kitchen - Testing harness tool, launches instance, runs config management code, runs test suite. Made by the chef guys

• Molecule - Similar to test kitchen, but written specifically for ansible

Full Disclosure: I'm the author of goss

UPDATE: InSpec 4.x or above uses a mixed commercial / open source license - see comments.

StackExchange DevOps Q#98, answer score: 24