snippetsqlgitMinor
How to create staging servers for multiple git branches?
Viewed 0 times
branchescreateserversstagingformultiplehowgit
Problem
I have to create a new staging process for our development and testing.
At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may be database evolution scripts (straight SQL) that must be run, as well as evolution scripts from the backend for heavier processing (essentially these are HTTP routes that must be called within the app using admin credentials that perform database migrations and other changes that would be too difficult/not possible to script in the aforementioned plain SQL evolution scripts).
Our live DB is moderately sized ~ 4.2 GB. We have a brand new Dell PowerEdge server that is ready to be set up and is at my disposable.
I would love advice on the following questions, as well as to know how more experienced DevOps would approach this:
-
How can I run several different branches on the staging server? These branches pop up and disappear frequently as they pass QA and are merged into master and are released.
-
How would I set up the DB evolution system to make sure it always has appropriate DB for each branch? Each branch may make modifications to the DB in different ways that are not necessarily compatible with each other until they're merged.
-
How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?
Would love any more input as I'm a bit lost on how to set this all up. Current workflow is difficult for everyone involved: Devs have a completely isolated local copy of the app running locally and QA has 3-4 laptops in rotation that act as their staging "servers"
At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may be database evolution scripts (straight SQL) that must be run, as well as evolution scripts from the backend for heavier processing (essentially these are HTTP routes that must be called within the app using admin credentials that perform database migrations and other changes that would be too difficult/not possible to script in the aforementioned plain SQL evolution scripts).
Our live DB is moderately sized ~ 4.2 GB. We have a brand new Dell PowerEdge server that is ready to be set up and is at my disposable.
I would love advice on the following questions, as well as to know how more experienced DevOps would approach this:
-
How can I run several different branches on the staging server? These branches pop up and disappear frequently as they pass QA and are merged into master and are released.
-
How would I set up the DB evolution system to make sure it always has appropriate DB for each branch? Each branch may make modifications to the DB in different ways that are not necessarily compatible with each other until they're merged.
-
How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?
Would love any more input as I'm a bit lost on how to set this all up. Current workflow is difficult for everyone involved: Devs have a completely isolated local copy of the app running locally and QA has 3-4 laptops in rotation that act as their staging "servers"
Solution
1) How can I run several different branches on the staging server?
Docker
2) How would I set up the DB evolution system to make sure it always has appropriate DB for each branch?
This depends on how much you expect your DB to scale. You can get pretty crazy with methods to clone database data but typically you will want a master copy that you do not alter until your code is released into production - and keep good backups when you do. While you infrastructure can be immutable and disposable, your data is not. You can only emulate disposability with you data. 4.2 GB really isn't that much to copy. You could make a script to spawn a new database instance for each build to operate on, then discard it once testing is complete.
3) How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?
You could consider using something like git hooks to trigger a build, force a code checkout or begin spawning a container and cloning your database. You could make an API call to a build automation system like Jenkins and/or use it to kick off a configuration management system like Puppet, Chef, Salt Stack, Ansible or something else. This would be more like an auto-pull than an auto push.
From the wording of your question, it's clear you are thinking of a mutable infrastructure, but consider using immutable testing deployments instead.
Docker
2) How would I set up the DB evolution system to make sure it always has appropriate DB for each branch?
This depends on how much you expect your DB to scale. You can get pretty crazy with methods to clone database data but typically you will want a master copy that you do not alter until your code is released into production - and keep good backups when you do. While you infrastructure can be immutable and disposable, your data is not. You can only emulate disposability with you data. 4.2 GB really isn't that much to copy. You could make a script to spawn a new database instance for each build to operate on, then discard it once testing is complete.
3) How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?
You could consider using something like git hooks to trigger a build, force a code checkout or begin spawning a container and cloning your database. You could make an API call to a build automation system like Jenkins and/or use it to kick off a configuration management system like Puppet, Chef, Salt Stack, Ansible or something else. This would be more like an auto-pull than an auto push.
From the wording of your question, it's clear you are thinking of a mutable infrastructure, but consider using immutable testing deployments instead.
Context
StackExchange DevOps Q#1568, answer score: 7
Revisions (0)
No revisions yet.