Blue Green Deployment DB transactions management

Imagining I have 2 sets of environment(Blue and Green)

+
           |
           |
           |
           v
   +-------+---------+
   |     Proxy/LB    |
   |                 |
   +--+-------------++
      |             |
+-----++          +-+----+
|      |          |      |
|APP v1|          |APP v2|
|Blue  |          |Green |
+--+---+          +--+---+
   |                 |
+--+---+          +--+---+
| DB   |          | DB   |
| Blue | DB Sync  | Green|
|            |
+------+          +------+

The version Blue is currently running and green is deployed and all DB replication completed. Now I want switch traffic to green environment. In this case, what is the best way to manage the DB transactions?(I have one thought below)

• Forward the new traffic to green and wait for existing blue's environment's transactions complete. Once all are done, switch traffic completely to green(UPDATE: Can I achieve this with Nginx?)

If above is true, I'd have to do little extra work to forward only new traffic to green.

Or what is the best way of doing Blue Green deployment in this case?

PS:

• We are using Nginx proxy

• All servers are bare metal, no cloud!

• 2 standalone DB setups(Postgresql with master-slave)

As per Martin Flowers' definition of Blue-Green deployment


The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.

Steps

You've said that DBs are synced using master-slave topology.
So, to switch traffic from one instance to another, we have to make steps:

• Switch web-traffic from APP v1 to APP v2

• Shutdown deprecated DB Blue

• Promote DB Green to master mode

• Start new DB Blue v2 in slave mode

(you may ask - why shutdown DB Blue instead of using master-master replication or other ways to sync DB Blue v1 with DB Green in master mode. See explanation in DB replication)

Graceful switch web-traffic

To switch web-traffic from APP v1 APP v2 - just use nginx reload - it will graceful end all working connection and move traffic from App v1 to App v2
There are many scripting solutions for switch blue/green using shell/python scripts

E.g.:

-
meappy/docker-nginx-blue-green: Docker blue green upstreams for nginx with ngx_http_perl_module

-
Simple 0-Downtime Blue Green Deployments: Python simple 0-Downtime Blue Green Deployments example

The logic is simple: add green upstream in nginx config, use it as default and send reload to nginx:

Sample nginx.conf for reverse proxy/load balancer

http {

upstream appv1 {
    zone appv1 64K;
    server 10.10.0.1:80;
}

upstream appv2 {
    zone appv2 64K;
    server 10.10.0.3:80;
}

server {
...
location / {
    proxy_pass http://$appv1;
}
}

}

Replace proxy_pass http://$appv1; with proxy_pass http://$appvv;

And then

nginx -s reload

Stop DB Blue

systemctl stop postgresql

Promote DB Green to master mode

• Trigger the promotion using your defined trigger in $PGDATA/recovery.conf

touch $PGDATA/failover

• Remove failover trigger from Green DB

cd $PRIMARY_DATA
rm -f recovery.* failover

• Make sure that hot stanby mode is on on Green DB in master mode

cat postgresql.conf | grep '#hot_standby = on'

DB replication

Make a backup of master Green DB

psql -c "SELECT pg_start_backup('Streaming Replication', true)" postgresql://postgres@GreenDB/postgres

Send a backup to Blue DB

rsync -a $PG_DATA_Green_DB/ $BlueDB_IP:$PG_DATA_Blue_DB/ --exclude postmaster.pid --exclude postmaster.opts

See also

• Evolution of Fault Tolerance in PostgreSQL

DB Migration (More complex but more robust database migration scheme)

You may use more complex scheme for migration using Ansible and Pglupgrade tool:

See for details:

-
Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part I)

-
Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part II)

P.s some definitions.

Just to...


... sort the buyers from the spyers, the needy from the greedy...

And blue/green deployment vs canary deployment vs A/B test

Definitions, kinds of deployments and tests

(see details in these answers )

-
Blue-Green Deployment - When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.

-
A/B Testing - Two versions of an application are running at the same time. A portion of requests go to each. Developers can then compare the versions.  

-
Canary Release - A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.

-
Feature flagging - The action of "configuring" (cold, or even hot) which functionality is (not)available for which (group) of users. If you also do something like "feature flagging" you can deploy first, measure soundness of your release in backwards compatibility/bug perspective, and release new functionality gradually to different users, or vice versa (scale down or even rollback functionality and/or binaries). Feature flagging allows for splitting availability of functionality from deployment of binaries, and gives much more fine-grained decision making then only "deploy/rollback"

P.P.S Blue/Green vs Canary`

source

-
Both blue-green and canary releases solve the same purpose

-
Although both of these terms look quite close to each other, they have subtle differences. One put confidence in your functionality release and the other put confidence the way you release.

Blue-Green Deployment


When deploying a new version of an application, a second environment is created. Once the new environment is tes

StackExchange DevOps Q#11173, answer score: 6