How do I make read only mount in Docker container writable?

When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I mount it read/write, the parallel processes will fight each other and I will need to be able to clean after them, there is also issue with privilege escalations on those files. If I mount it read/only then I cannot run processes that write files inside the directory.

Docker suggest to build a new container with a COPY of the repository, save it as new layer and then start several containers from this new image. Having to make a copy of megabytes of data every time you want to run tests is making it take too long though.

How can I solve this efficiently?

Solution to this problem is to create an overlay filesytem over a read-only mount, but if you try to do it directly, overlay will refuse to put upper and work directories on another overlay filesystem. The trick is to create a tmpfs for the upper and work directories like this:

Create a script called run-in-c.sh

#!/bin/bash
NAME=$1
shift
HOSTNAME=dock${NAME}
CONTAINER=dc-${USER}-${NAME}
REPOSITORY=${HOME}/repository
BASEIMAGE=hub.docker.io/my-org/my-base-container
OVERLAY=/mnt/overlay
LOWERDIR=/mnt/lower
UPPERDIR=${OVERLAY}/upper
WORKDIR=${OVERLAY}/work
TARGET=/mnt/repository
# PRIVILEGED="--privileged"
PRIVILEGED="--cap-add SYS_ADMIN"
docker container create --name $CONTAINER $PRIVILEGED --hostname $HOSTNAME --volume ${CO}:${LOWERDIR}:ro $BASEIMAGE
docker container start $CONTAINER
docker container exec $CONTAINER mkdir -p $OVERLAY
docker container exec $CONTAINER mount -t tmpfs tmpfs $OVERLAY
docker container exec $CONTAINER mkdir -p $WORKDIR $UPPERDIR $TARGET
docker container exec $CONTAINER mount -t overlay overlay -o lowerdir=${LOWERDIR},upperdir=${UPPERDIR},workdir=${WORKDIR} $TARGET
docker container exec -it --workdir $TARGET $CONTAINER $*
docker container stop $CONTAINER
docker container rm $CONTAINER

Then you can run commands in the container as:

run-in-c.sh test01 'cd dir && command args'

or simply get interactive shell with:

run-in-c.sh naming-stuff-is-too-hard bash

StackExchange DevOps Q#3872, answer score: 2