How can I manage secrets in .tf and .tfstate?

I would like to use the Terraform MySQL Provider to keep a list of mysql users and grants handy for creating new test environments. The .tf and .tfstate files both seem to want to store the MySQL passwords in plaintext.

Concerning .tf:

It is my understanding that .tf files live in revision control and are maintained by a team. How does that practice differ when secrets are in the .tf? It is possible to encrypt these values at all?

Concerning .tfstate:

I can store the .tfstate securely somewhere after running Terraform apply, but it would be preferable for this use case to not store it at all?

Terraform supports adding an additional file with variables during invocation.

documentation: https://www.terraform.io/intro/getting-started/variables.html#from-a-file

We are using that feature to provide a secrets.tfvars file on each invocation of Terraform. We also use a script to wrap the command so that its invocation is consistent, and all team members avoid having to make the same mistakes. The wrapper synchronizes .tfstate with S3 before an execution, and pushes .tfstate back to S3 at the end. I also hear of people doing the same thing with state stored in Consul, even adding a kind of semaphore in consul to prevent two people from starting Terraform at the same time.

When you avoid setting a default value in a variables.tf file, it forces the user to input the value. It can be either entered manually or using the -var-file command option like described above. Not setting a default on your secrets is a good way to enforce changes that require a change in secrets.

The secrets.tfvars file is a symbolic link to one of the files with secrets which are not stored in version control. We have several, one per environment, like so secrets-prod.tfvars, secrets-dev.tfvars, secrets-stg.tfvars, etc...

An even better practice would be to generate these secrets files during the wrapper script based on data in Vault or some other way to share secrets. Since currently when the format of secrets changes, or secrets themselves, we need to communicate it to the team outside the version control channel - and this doesn't always work well, to be honest. But secrets do change infrequently.

StackExchange DevOps Q#79, answer score: 30