Deploying a Kubernetes cluster for production on multiple clouds

I want to install a highly available kubernetes cluster (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).

I'm looking for a non-vendor specific option to do it.

I'm aware of several options like

• Kops

• Juju

• Other Github projects I found...

I was wondering if anyone has a good automated example of doing this with such tools like Terraform, Ansible or others.

The HA is a must as this if for production use!

At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using kops along with Terraform (kops generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.

Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.

The first thing in order to create the cluster is to generate the Terraform configuration with kops (you could directly apply the changes in AWS by directly using kops, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:

kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=. \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc= \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform

This creates the Terraform configurations (in the directory specified by the --out flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:

• all necessary subnets, security groups and route tables

• IAM roles

• AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)

• all EC2 instances and EBS volumes

• Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.

kops uses kubectl internally to create the cluster and all necessary configurations in Kubernetes.

After that, you could terraform apply the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.

Later on, you may need to edit you cluster. There's a command: kops edit cluster that let you do that. It's all configurations in yaml format. I won't get into specifics but after some changes you may need to kops rolling-update to apply them.

Hope I've been clear.

Regards.

StackExchange DevOps Q#1624, answer score: 6