Use a private RUN cache between builds in BuildKit

Posted on by Yuki Nakamura

TL;DR

This post provides hands-on tutorial on how to use a private RUN cache that is defined by RUN –mount=type=cache, sharing=private in Dockerfile. Here is a snippet of Dockerfile.

RUN --mount=type=cache,id=private-cache,target=/root/.cache,sharing=private \
  echo $(date):" private1 is writing..." && \ 
  echo $(date)": Hello from private1" >> /root/.cache/private.log && \
  sleep 10 && \
  echo "--- private.log ---------" && \
  cat /root/.cache/private.log && \
  echo "------------------------" && \
  echo $(date):" private1 finish writing"
A private RUN cache

RUN cache has three sharing types (shared, private and locked) . In the two previous posts, I explained shared and locked types’ behaviors. In this post, we will try to use a private type.

RUN –mount=type=cache
  1. Clone the repository from GitHub
  2. Create a private RUN cache
  3. The private cache behavior
    1. Multiple writers
    2. The first cache was deleted?
    3. Appendix: Which cache is used in a next build?
  4. Wrap up

Clone the repository from GitHub

I already prepared contents for hands-on, so please clone it from my repository named buildkit–101.

git clone https://github.com/yukinakanaka/buildkit-101.git

Then, move to the following directory.

cd buildkit-101/run-cache/private

Create a private RUN cache

Create a private RUN cache

Dockerfile.private1 contains RUN –mount=type=cache,id=private-cache,target=/root/.cache,sharing=private. It will create a RUN cache during building a DockerImage.

 cat Dockerfile.private1 
FROM debian:12.4-slim
RUN mkdir -p /root/.cache
COPY rebuild-trigger .

RUN date
RUN --mount=type=cache,id=private-cache,target=/root/.cache,sharing=private \
  echo $(date):" private1 is writing..." && \ 
  echo $(date)": Hello from private1" >> /root/.cache/private.log && \
  sleep 10 && \
  echo "--- private.log ---------" && \
  cat /root/.cache/private.log && \
  echo "------------------------" && \
  echo $(date):" private1 finish writing" 

RUN date

Let’s build a DockerImage from Dockerfile.private1!

docker build --progress=plain -f Dockerfile.private1 .

In the build log, you can see that the build process wrote a log in /root/.cache/private.log. The file private.log was stored in a cache named private-cache!

#9 [stage-0 5/6] RUN --mount=type=cache,id=private-cache,target=/root/.cache,sharing=private   echo $(date):" private1 is writing..." &&   echo $(date)": Hello from private1" >> /root/.cache/private.log &&   sleep 10 &&   echo "--- private.log ---------" &&   cat /root/.cache/private.log &&   echo "------------------------" &&   echo $(date):" private1 finish writing"
#9 0.077 Sun Mar 10 05:49:38 UTC 2024: private1 is writing...
#9 10.08 --- private.log ---------
#9 10.08 Sun Mar 10 05:49:38 UTC 2024: Hello from private1
#9 10.08 ------------------------
#9 10.08 Sun Mar 10 05:49:48 UTC 2024: private1 finish writing
#9 DONE 10.1s

The private cache behavior

Multiple writers

Multiple processes cannot write the same private cache at same time. When there are multiple writers, the new private cache will be created.

Multiple processes cannot write the same cache at same time

Let’s check the private cache behavior by running two build processes at the same time!

echo $RANDOM > rebuild-trigger 
docker build --progress=plain -f Dockerfile.private1 . 1>private1.log 2>&1 &
sleep 1
docker build --progress=plain -f Dockerfile.private2 . 1>private2.log 2>&1 &

Please check the logs in private2.log. You can see only private2’s log, you cannot see the private1’s log that was written in previous step. This means that a new cache was created!

$ cat private2.log   
...
#9 10.09 --- private.log ---------
#9 10.09 Sun Mar 10 05:50:09 UTC 2024: Hello from private2
#9 10.09 ------------------------
...

Please check the logs in private1.log, too. You can see private1 uses the same cache in the previous step.

$ cat private1.log   
...
#9 10.28 --- private.log ---------
#9 10.28 Sun Mar 10 05:49:38 UTC 2024: Hello from private1
#9 10.28 Sun Mar 10 05:50:09 UTC 2024: Hello from private1
#9 10.28 ------------------------
...

The first cache was deleted?

We found the new cache was created. Then, what about the first created cache? Let’s check it.

echo $RANDOM > rebuild-trigger 
docker build --progress=plain -f Dockerfile.private1 . 1>private1.log 2>&1 &
sleep 1
docker build --progress=plain -f Dockerfile.private2 . 1>private2.log 2>&1 &
The first cache is still there

You can find the first private.log in private1.log or private2.log. In my case, I found it in private1.log. This means the old cache is not deleted!

$ cat private1.log   
...
#9 10.26 --- private.log ---------
#9 10.26 Sun Mar 10 05:49:38 UTC 2024: Hello from private1
#9 10.26 Sun Mar 10 05:50:09 UTC 2024: Hello from private1
#9 10.26 Sun Mar 10 05:50:48 UTC 2024: Hello from private1
#9 10.26 ------------------------
...

Appendix: Which cache is used in a next build?

I tried to find a rule for which cache is used preferentially, but I was not able to find it. Sometimes, the old cache is used, and sometime the new cache was used. Also, its priority sometimes was changed after multiple writers ran.

Wrap up

In this post, I explained the behavior private-type RUN cache between builds. These are features of private-type RUN cache. 🐳

  • A new cache is created when some build process is using existing cache.
  • The first (old) cache is not deleted.

Please check bellow if you’re interested in other type of cache.

Unknown's avatar

Published by Yuki Nakamura