Debug Docker caching issues

I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.

The docker cache needs the following:

• Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and

• The previous image needs to still be on the build host (not pruned) and

• The same previous layer and either

• The same command being run (including the same environment/args) or

• The same hash on the files being copied

If you see the cache being busted on a COPY or ADD command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a docker image history command, e.g.:

$ docker image history golang:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB

You can include the --no-trunc option to get a full line. The key piece of data relevant to your question from the above image is in the ADD step where you see file:0eb5ea35741d23fe3…. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.

StackExchange DevOps Q#8780, answer score: 3