Build Docker images with DinD and BuildKit

I use GitLab's pipelines to describe the deployment process. And, as part of it, I build Docker images. For that purpose, I use Docker-in-Docker to build those images (i.e. download the Docker-in-Docker image and do all the Docker-stuff, inside of it).

I also set to use Docker Buildkit features. And one of them is --cache-from option (which decreases build time). Which seems doesn't work with Docker-in-Docker (at least, for me).

Here's the pipeline's config:

stages:
  - build
  - purge
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_BUILDKIT: 1
  LATEST: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_REF_SLUG}
  IMAGE_COMMIT_TAG: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}

services:
  - docker:dind

build:
    stage: build
    image: docker:stable
    script:
        - docker login -u gitlab-ci-token -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
        - docker build --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
        - docker push ${IMAGE_COMMIT_TAG}
        - docker push ${LATEST}

The log output shows the BuildKit is engaged, but there's nothing about using of 'CACHE'. And, the result of it -- 'build' stage takes much more time.

Please, your suggestions, ideas?

Caching of layers with buildkit in an external registry requires an extra step or two depending on how you want to cache your layers. The easy option is to include a build arg that enables the inline cache:

docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

Note that the inline cache only caches layers for the target stage that was pushed in the image, so other states in a multi-stage build would need to be built and cached separately, or rebuilt without caching, neither of which is ideal.

You can also cache to a local file, or push the cache to a different registry image rather than inline with the image you pushed. Unfortunately the standard docker build CLI doesn't have access to all the buildkit flags to enable this. Instead, you can install buildkit directly or use buildx which is a CLI plugin for managing buildkit. You can install buildx separately but in current releases it's available with experimental CLI options that can be enabled with export DOCKER_CLI_EXPERIMENTAL=enabled. I believe you'd need to create a container based builder for all of the buildkit options, which can be done with:

docker buildx create --use --driver docker-container --name local ${DOCKER_HOST:-unix:///var/run/docker.sock}

Then you should be able to run something like:

docker buildx build --cache-from ${IMG_CACHE} --cache-to ${IMG_CACHE} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

More documentation on the buildkit caching is available at: https://github.com/moby/buildkit#export-cache

More details on buildx is available at: https://github.com/docker/buildx

StackExchange DevOps Q#11175, answer score: 4