How to investigate a main process that has died in a docker container?

Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.

docker exec -ti bash only works on running containers, once it finishes, the bash prompt terminates as well.

With docker start you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.

We can do docker commit, then docker run on the new image with a different command, but I'm wondering if there are any other alternatives.

Note: docker logs just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.

General ways to track why a process in Linux failed are good. One such way is to run a process using strace which will tell you the system calls process did and usually point to the reason for a failure.

You can create a Dockerfile that looks something like this:

FROM original_image

RUN apt-get -y update && apt-get install -y strace

# build with `docker build -t debug_version`

Then run your new image using docker run debug_version strace original_cmd.

For processes that fork children (and then die) you want to run strace with the -ff option. You can also map some file using Docker data volumes and use the -o option from strace to write to it. But in general strace will leave output on stdout, which is readable using docker log.

Related Q: Linux process terminates mysteriously

StackExchange DevOps Q#419, answer score: 9