snippetkubernetesMinor
How to manage 10+ team environments with Helm and Kubernetes?
Viewed 0 times
withkubernetesteammanagehelmhowandenvironments
Problem
I'm looking for feedback on how we can improve our team environment and staging deployment process.
We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:
All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (
The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The
For each environment that we deploy into, we clone the repo, change the values of the
This results in us basically creating a series of namespaces such as
We are currently hitting a few main roadblocks.
1. It's hard to deploy to multiple clusters or new clusters.
2. It's hard to get a single source of truth for the configurations.
3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.
I have come up with three ways of moving forward, all of which "feel wrong" to me.
Are we doing
We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:
All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (
deploy.sh, buildSecrets.sh, and buildConfigMaps.sh) and some json configuration files. (namespaceCharts.json, secrets.json, configmaps.json)The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The
namespaceCharts.json defines which charts need to be deployed into each namespace grouping.For each environment that we deploy into, we clone the repo, change the values of the
secrets.json and configmaps.json and then run the scripts passing in our base namespace name. (For example, the team name)This results in us basically creating a series of namespaces such as
teamA-frontend and TeamA-backend each of which contains the right microservices with the connection strings to the correct DB etc.We are currently hitting a few main roadblocks.
1. It's hard to deploy to multiple clusters or new clusters.
2. It's hard to get a single source of truth for the configurations.
3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.
I have come up with three ways of moving forward, all of which "feel wrong" to me.
- Store all the environment configurations in separate "configurations" git repo.
- Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.
- Use ansible or something similar to store the scripts and configurations for each environment.
Are we doing
Solution
My 2 cents. We have similar requirement like you do. We end up having multiple values files in the form of
We also have some shell scripts to deploy to the namespace. Something like this deploy.sh --namespace app.
It is not without its problem, the number of files are steadily growing. But so far we are ok with it.
values-.yaml and it should come with the defaults set to values.yamlWe also have some shell scripts to deploy to the namespace. Something like this deploy.sh --namespace app.
It is not without its problem, the number of files are steadily growing. But so far we are ok with it.
Context
StackExchange DevOps Q#6641, answer score: 1
Revisions (0)
No revisions yet.