HiveBrain v1.2.0
Get Started
← Back to all entries
snippetdockerMinor

How to Shutdown Docker Stack on Container Failure

Submitted by: @import:stackexchange-devops··
0
Viewed 0 times
containerstackdockershutdownhowfailure

Problem

Say I have several docker services using swarm defined as such:

version : '3'
services:
    server:
        image : my_server:latest
        deploy: 
            replicas: 5
            restart_policy: any
    database:
        image : my_db:latest
        deploy: 
            replicas: 5
            restart_policy: any
    logs:
        image : my_logger:latest
        deploy: 
            replicas: 1
            restart_policy: any


I then deploy this as a docker-stack as such. The services are server, database and logs. I make great use of the network abstractions provided by docker swarm, but my failure-case is a bit undockerish.

In this case when all containers in database fail, then I want the entire stack to be shutdown or otherwise made unavailable. Is there any built-in way to accomplish this task? Does the concept of a stack exist in a way for the individual services to access?

This question can be alternatively answered with a generalization : "How, with built-in docker features, can I be made aware of container failure"? Because I'm assuming there is no way to do exactly what I am asking.

Solution

If you don't want the containers to restart, you can set the condition: to none. See https://docs.docker.com/compose/compose-file/#restart_policy

On the swarm manager, you could use docker stack ps to check the status of the containers on a regular basis and shut down the stack if the database containers aren't running (using cron or whatever). For example:

docker stack ps --filter "desired-state=Running" your-stack |grep database || docker stack rm your-stack.

If containers are not able to restart, they shouldn't show up in the list once they fail, and once none are found, you can remove the stack.

Context

StackExchange DevOps Q#2490, answer score: 1

Revisions (0)

No revisions yet.