I recently came across Staying safe with .NET containers on Hacker News. It’s an excellent post that goes into how the .NET Docker image publishing team thinks about:

  • The pedigree and provenance of the .NET Docker images they publish
  • The process by which they build and publish the images
  • Vulnerabilities (read: CVEs) in the base images they build from, and in the images they publish

With the devil often being in the details, I want to challenge a particular assumption in the article regarding Docker image composition. The article suggests that the “history” that is recorded while building a Docker image is a trustworthy record, which in this post I will demonstrate it is not – it is simple metadata that can be tampered with.

I reached out to Richard, the author of the Microsoft article, before putting this post together. He encouraged me to dig deeper into my thoughts and feelings and to share my reasoning for the benefit of all.

I’m certainly not having a dig at Microsoft or Richard. The article is a fantastic and thoughtful read, and if you haven’t read it I highly encourage you to do so.

Update 2021-02-16: Richard has updated the article. A section titled “Critical note” now acknowledges the fact that Docker image history is not necessarily trustworthy and that the guidance regarding container-diff shouldn’t be used to draw security-critical conclusions. Thanks Richard, it was a pleasure chatting to you about this one!

tl;dr

You should not trust the “history” of a potentially untrustworthy Docker image. The history is not a trustworthy record that is guaranteed to reflect the contents of the image. The history is metadata and it can be edited. If you are using the image history as a proxy for its authenticity, you are vulnerable to a certain type of supply chain attack.

This is not a vulnerability in Docker, it’s simply a misunderstanding and a misuse of one of its features.

If you need to be able to assert the authenticity of Docker images you are building or consuming, you may be able to use Docker Content Trust (i.e. image signing) to achieve your goals.

The article

My reading of the section titled “Dockerfiles” gave me the following impression:

  • The team puts a lot of thought and care into how they craft their Dockerfiles
  • The team publishes their Dockerfiles. For example, here is the Dockerfile for amd64 .NET 5.0 built atop Alpine 3.13 (In case you missed it, “.NET” is the new name for the old “.NET Core”)
  • The team builds the published .NET images from the very same set of published Dockerfiles
  • Thus, customers can safely build the images themselves using the published Dockerfiles, or they can pull the published images from Microsoft’s registry as a convenience mechanism, and can rest assured that they should get the same result either way
  • Furthermore, customers can use Google’s container-diff tool with the --type=history flag to compare a locally-built image with an image pulled from Microsoft’s registry to verify that the image on Microsoft’s registry was built using the same Dockerfile. The article notes that the image digests will not match. This is due to the fact that Docker images currently cannot be built deterministically.

It’s this last point that jumped out at me. In this post, I’ll dive into the impression that one might get from reading the article, and why container-diff --type=history is not a suitable technique for validating that a Docker image was built from a particular Dockerfile.

An aside: Build determinism and “Reproducible Builds”

Before we get too far, a short discussion of build determinism and the idea of “Reproducible Builds” may be useful.

“Building” or “compiling” something is the process of taking a set of artefacts (Sometimes source code) and transforming them into a distributable, executable, usable artefact. In the context of Docker images, this is done using docker build which transforms a Dockerfile into a Docker image. By my own understanding, the docker build command does something similar to the following (Assuming a simple Dockerfile with only one FROM command):

  1. Start with the base image (e.g. Debian, Ubuntu, Alpine) referenced by the FROM command in the Dockerfile
  2. Run through the Dockerfile top-to-bottom, executing each command and capturing the “delta” effect that each command has on the image being built
    • In this context, “command” means a command specified by the Dockerfile Reference. For example – ADD adds a file from the local filesystem or a URL to the image being built, while RUN runs a shell command within the image being built.
  3. Arrange the pile of “delta” images into a stack and decorate the pile with some metadata

The resulting stack of delta images is the output artefact (the “image”) and can be executed on the local machine using docker run or can be pushed to a Docker image registry (docker push) to be pulled by others (docker pull).

A “deterministic” build would mean that if two people were to perform the build process at two different times on two different hosts, the output would be byte-for-byte identical. Deterministic builds are very attractive, as they enable Reproducible Builds.

Purely from my own experience and reasoning, I think it’s not easy or even practical to build Docker images in a reproducible way. For example:

  • If the Dockerfile specifies that packages should be updated or installed from an OS repository (e.g. via apt for Debian/Ubuntu or via apk for Alpine) then the build will vary if the OS publishes updated packages between two different docker build executions
  • If the installation of a package causes varying install-time data to be written (e.g. an installation ID or an SSH host key) then the build will vary each time docker build is run
  • If docker build is done at two different points in time, then any command that causes a state change to the filesystem will result in different filesystem timestamp data being captured in the layer diffs which will cascade through to what would otherwise be a great point of comparison – the layer hashes

Note that these are just a few reasons why I think that Docker images are difficult or perhaps even impossible to build deterministically. There may well be ways to mitigate some or all of these, and there are likely other ways that non-determinism can “leak” into the building of a Docker image. The Microsoft post notes that people are working on deterministic Docker image building, so it may one day become practical to deterministically build Docker images.

With that out of the way, the reason why deterministic builds are attractive is that they enable Reproducible Builds. Once something becomes deterministically buildable, if the builder commits to taking any care needed to ensure that their buildable thing is built deterministically, we gain something called Build Reproducibility. This allows anyone in the world to do the following:

  1. Obtain the source artefacts for the buildable thing
  2. Obtain a copy of the published “built thing”
  3. Independently build the buildable thing
  4. Verify that the obtained “built thing” (From 2) is a byte-for-byte copy of the independently built “buildable thing” (From 3)

Of course, at this stage, the person doing the experiment could simply use their locally-built identical copy of the buildable thing anyway. As Debian notes, the simple fact that someone can do this experiment gives us some interesting security benefits:

  • An attacker who is in a position to compromise the supply chain by tampering with published “built things” is now disincentivized to do so since they know their tampering can be detected by anyone
  • If an attacker is still bold enough to attempt their tampering, if they are detected by someone doing the “build locally and compare” experiment, the compromise can be identified and remediated

If you’d like to generally know more about build determinism and Reproducible Builds, the types of challenges that need to be solved to make builds deterministic, and the security and non-security benefits of Reproducible Builds, check out the Reproducible Builds project as well as Debian’s ReproducibleBuilds project.

Finally, I know enough about Docker and the building of Docker images to be dangerous, but I’m certainly not a Docker expert. Without question there are better resources out there regarding Docker image build determinism, and if you’re interested I encourage you to seek them out. For now, my “thoughts and feelings” should be enough to explain why I think the Microsoft post said something that it did, and why its advice on the matter is subtly misleading.

Using Docker image history as a proxy for build authenticity

Putting our attention back on Staying safe with .NET containers, the author explains that users of the published .NET Docker images are encouraged to consider whether an image obtained from the Microsoft registry is genuine.

Imagine you pull and inspect a .NET image (from our registry) and then rebuild the same image from the Dockerfile we’ve shared as its apparent source, on your own machine. You get the same result (I’ll define that shortly). That’s comforting. It means that using official .NET images is just a convenience as you can build them yourself. But what happens if you get a different result. That’s concerning, particularly if no explanation is provided. What are you to think? Your mind races. The difference could be the result of something nefarious or accidental. Only an investigation could help answer that question, and who has time for that?

I am confident that Microsoft takes the security of their .NET image build pipeline and Docker image registry extremely seriously, but it’s great to see them suggesting that users should still ask the question – how can I verify that a .NET image from the Microsoft registry is a genuine, safe artefact?

The author goes on to say:

The following workflow demonstrates how to compare a registry image with a locally built one (both from the same source Dockerfile) using the Google container-diff tool:

This result tells you that the Dockerfiles used to build the two images are the same. That’s — as far as I’m aware — the most straightforward test to validate the fidelity of a registry image with its apparent source. The test passed. Comparing image digests won’t work; they will not match with normal practices.

It’s the first part of this statement that concerns me.

To put it in other words, the author is suggesting:

  • The user should obtain a published .NET Docker image
  • The user should obtain the Dockerfile from which the .NET Docker image publishing team builds the published Docker images
  • The user should build their own copy of the .NET Docker image
  • The user should use container-diff tool with the --type=history option to “compare the images”
  • If the container-diff tool reports that no Docker history lines differ between the published image and the locally-built image, the user should rest assured that the images were built with the same Dockerfile

Since Docker image builds are not deterministic, the author is suggesting that the Docker image history be used as a proxy for the autenticity of an otherwise untrustworthy image.

We’ll get to why I don’t think this thinking is correct. But first, some more background.

Docker image history

As explained previously, the process of building a Docker image runs through a Dockerfile top-to-bottom and runs each command, capturing the deltas at each stage into a stack of diffs. The build process also decorates each diff in the stack with some metadata, including the command used to generate that layer.

For example, given the following trivial Dockerfile:

We can build the image:

And inspect its history:

We can see in the output of docker history that the command executed at each stage of the docker build process is recorded against each layer in the stack.

container-diff –type=history

The container-diff tool isn’t terribly descriptive about how the --type=history differ works. The README.md says:

The history analyzer outputs a list of strings representing descriptions of how an image layer was created.

The code says:

(Note that getHistoryList() seems be digging out some field called layer.CreatedBy. Note also that the output of docker history justinsteven/foobar above has a column named “CREATED BY” with the list of commands that created each layer)

Based on these descriptions, and based on my usage of the tool, container-diff with the flag --type=history seems to compare the command-by-command layer-by-layer history of Docker images.

Modifying Docker image histories

I will show that Docker image histories (which, once more for those in the back, are simply metadata) can be trivially modified. I’ll do so by creating a justinsteven/baddotnet-sdk:5.0-alpine Docker image from a modified Dockerfile, and will then edit the history so that it matches the history of the genuine mcr.microsoft.com/dotnet/sdk:5.0-alpine image. The image will appear at a glance to be built from the genuine Dockerfile and will even satisfy the comparison approach suggested by Microsoft – that is, the use of container-diff with the flag --type=history will report that there is no difference between the commands in each image’s history.

Environment

I will be using an Ubuntu 20.04 VM with Docker v20.10.3 installed as per https://docs.docker.com/engine/install/ubuntu/

Installing container-diff

I have installed the binary release of container-diff v0.15.0 as per https://github.com/GoogleContainerTools/container-diff#linux

Note that I wouldn’t normally yolo-curl a binary to /usr/local/bin/ but this is a disposable VM :^)

Building a modified .NET image as justinsteven/baddotnet-sdk:5.0-alpine

Obtain the source Dockerfile:

Modify the last line of the Dockerfile and “poison” the dotnet binary:

(This Dockerfile, when built, will generate a Docker image with a /usr/bin/dotnet executable file that will simply echo a fun message.)

Build the image as justinsteven/baddotnet-sdk:5.0-alpine:

Run the image and execute the dotnet binary:

Edit the Docker image history of justinsteven/baddotnet-sdk:5.0-alpine

To edit the Docker image history I will:

  1. Save the Docker image to a .tar file
  2. Unpack the .tar file
  3. Edit the metadata that specifies the image history
    • In particular, I will remove the part of the command that I tacked on to the last line of the Dockerfile
  4. Repack the .tar file
  5. Load the .tar file over the top of the existing Docker image in our local cache

Save the image:

Extract the saved image:

Modify the <sha256>.json file in the root of the extracted image to rewrite the history of the image:

Repack the .tar:

Load the .tar. Doing so will modify the justinsteven/baddotnet-sdk:5.0-alpine image in our local cache.

Inspecting the history and running container-diff

We can now use docker history to inspect the history metadata of the local justinsteven/baddotnet-sdk:5.0-alpine image. Note that viewing the history of images with long commands requires the --no-trunc flag to show all of each line, and be warned that the output is a little unwieldy.

The column of Image IDs is now littered with the value <missing> but in my experience this is par for the course for a Docker image obtained from a remote registry or imported from a .tar file.

We’re now in a position to use container-diff with the --type=history flag to compare our justinsteven/baddotnet-sdk:5.0-alpine image with Microsoft’s mcr.microsoft.com/dotnet/sdk:5.0-alpine

container-diff tells us that no Docker history lines differ between the two images, and yet the justinsteven/baddotnet-sdk:5.0-alpine image is clearly a bit wonky:

We can even push the image to Dockerhub, remove it from our local cache, pull it down and repeat the exercise.

We can successfully pull justinsteven/baddotnet-sdk:5.0-alpine and execute the dotnet command within it to get our funny little message. container-diff still sees the history as being identical to the official mcr.microsoft.com/dotnet/sdk:5.0-alpine image.

This image is available on Dockerhub at the time of writing. I may remove it in the future, but if you get in quick you can pull it down and have a play.

So what?

The ability to modify Docker image histories means that histories should not be trusted, and should not be used as a proxy to determine the authenticity of a Docker image.

For this technique to be useful to an attacker, in the specific case of Microsoft’s .NET images, the attacker would need to:

  1. Be a malicious insider within Microsoft, or an external attacker, who can compromise the .NET image build pipeline or the Microsoft Docker image registry
  2. Publish malicious .NET images or replace published .NET images with malicious contents, using this technique to rewrite the image’s history and hide the tampering.
  3. Evade any behind-the-scenes spot checks that Microsoft are doing on their published .NET images, and any checks that the wider community is doing on the published .NET images
  4. Trust that Microsoft’s users are only performing naive checking of the Docker image history to authenticate the image

To pull this off would be no small feat, and I’m not suggesting that Microsoft’s .NET images have been tampered with. All I am suggesting is that, if they had, the guidance given in Staying safe with .NET containers is insufficient to detect conscientious tampering.

Image signing via Docker Content Trust (DCT)

Only just now did I wonder – “Is there a content signing mechanism for Docker images?” It turns out there is something called Docker Content Trust (DCT) which claims to allow image builders to sign images with a private key, so that consumers can verify the image authenticity. I haven’t heard of DCT before today so I don’t know how usable it is or how widely it’s used, but it may be useful for some builders or consumers of Docker images.

Conclusion

My advice to Microsoft is – please reconsider recommending that people use container-diff to verify the authenticity of a Docker image using its history as a proxy. The image history is metadata only, and can be modified by an image author or by anyone who can get their hands on any Docker image.

Update 2021-02-16: The article has been updated. A section titled “Critical note” now acknowledges the fact that Docker image history is not necessarily trustworthy and that the guidance regarding container-diff shouldn’t be used to draw security-critical conclusions.

My advice to users of Docker images is – don’t trust the image history of a potentially untrustworthy image. It can be modified.

I don’t have any advice for Docker or for Google (the authors of container-diff). However, it might be a good idea to remind users at the appropriate time that Docker image histories are not immutable records of the building of an image – they can be modified, and hence should not be trusted.

Reproducible Builds allow anyone to independently verify that a given built artefact is identical to a locally built copy. They discourage and make visible some types of supply chain attacks. Reproducible Builds require build determinism, which Docker images lack, but some people are reportedly working on it.