Troubleshooting permission issues when building Docker containers

Docker containers run by default as root. In general, it is not a recommended practice as it poses a serious security risk. This risk can be mitigated by configuring a non-root user to run the container. One of the ways to achieve this is to use the USER instruction in the Dockerfile. While running a container as a non-root user is the right thing to do, it can often be problematic as insufficient permissions can lead to hard to diagnose errors. This post uses an example node application to discus a few permission-related issues that can pop up when building a non-root container along with some strategies that can help troubleshoot this kind of issues.

Example

Let’s start with a very simple Dockerfile for a node application:

FROM node:16
WORKDIR /usr/app
COPY . .
RUN npm install
CMD [ "node", "/usr/app/index.js" ]

The problem with this Docker file is that any Docker container created based on this file will run as root:

To fix that we can modify the Docker file to create a new user (let’s call it app-user) and move the application to a sub-directory in the user home directory like this:

FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
COPY . .
RUN npm install
CMD [ "node", "index.js" ] 

Unfortunately, introducing these changes makes it impossible to build a docker image – npm install now errors out due to insufficient permissions:

Step 8/9 : RUN npm install
 ---> Running in a0800340b850
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /home/app-user/app/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/home/app-user/app/node_modules'
npm ERR!  [Error: EACCES: permission denied, mkdir '/home/app-user/app/node_modules'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/home/app-user/app/node_modules'
npm ERR! }
...

Inspecting the app directory shows that the owner of this directory is root and other users don’t have the write permission:

app-user@d0b48aa18141:~$ ls -l ~
total 4
drwxr-xr-x 1 root root 4096 Jan 15 05:48 app

The error is related to using the WORKDIR instruction to set the working directory to $HOME/app. It’s not a problem by itself – it’s actually recommended to use WORKDIR to set the working directory. The problem is that because the directory didn’t exist, WORKDIR created one, but it made root the owner. The issue can be easily fixed by explicitly creating a working directory with the right permissions before the WORKDIR instruction runs to prevent WORKDIR from creating the directory. The new Dockerfile that contains this fix looks as follows:

FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
COPY . .
RUN npm install
CMD [ "node", "index.js" ]

Unfortunately, this doesn’t seem to be enough. Building the image still fails due to a different permission issue:

Step 10/11 : RUN npm install
 ---> Running in 860132289a60
npm ERR! code EACCES
npm ERR! syscall open
npm ERR! path /home/app-user/app/package-lock.json
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, open '/home/app-user/app/package-lock.json'
npm ERR!  [Error: EACCES: permission denied, open '/home/app-user/app/package-lock.json'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'open',
npm ERR!   path: '/home/app-user/app/package-lock.json'
npm ERR! }
...

The error message indicates that this time the problem is that npm install cannot access the package-lock.json file. Listing the files shows again that all copied files are owned by root and other users don’t have the write permission:

ls -l
total 12
-rw-r--r-- 1 root root  71 Jan 15 02:03 index.js
-rw-r--r-- 1 root root 849 Jan 15 01:36 package-lock.json
-rw-r--r-- 1 root root 266 Jan 15 05:21 package.json

Apparently, the COPY instruction by default uses root privileges, so the files will be owned by root even if the COPY instruction appears the USER instruction. An easy fix is to change the Dockerfile to copy the files before configuring file ownership (alternatively, it is possible specify a different owner for the copied files with the --chown switch):

FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
COPY . .
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
RUN npm install
CMD [ "node", "index.js" ]

Annoyingly, this still doesn’t work – we get yet another permission error:

Step 9/10 : RUN npm install
 ---> Running in d4ebcec114cb
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/node_modules'
npm ERR!  [Error: EACCES: permission denied, mkdir '/node_modules'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/node_modules'
npm ERR! }
...

This time the error indicates that npm install tried creating the node_modules directory directly in the root directory. This is unexpected as the WORKDIR instruction was supposed to set the default directory to the app directory inside the newly created user home directory. The problem is that the last fix was not completely correct. Before, COPY was executed after WORKDIR so it copied the files to the expected location. The fix moved the COPY instruction so that it is now executed before the WORKDIR instruction. This resulted in copying the application files to the container’s root directory, which is incorrect. Preserving the relative order of these two instructions should fix the error:

FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
WORKDIR $HOME/app
COPY . .
RUN chown -R app-user:app-user $HOME
USER app-user
RUN npm install
CMD [ "node", "index.js" ]

Indeed, building an image with this Dockerfile finally yields:

Successfully built b36ac6c948d3

The application also runs as expected:

Debugging strategies

Reading about someone’s errors is one thing, figuring the errors out oneself is another. Below are a few debugging strategies I used to understand the errors described in the first part of the post. Even though I mention them in the context of permission errors they can be applied in a much broader set of scenarios.

Carefully read error messages

All error messages we looked at were very similar, yet each signaled a different problem. While the errors didn’t point directly to the root cause, the small hints were very helpful in understanding where to look to investigate the problem.

Check Docker documentation

Sometimes our assumptions about how the given instruction runs may not be correct. Docker documentation is the best place to verify these assumptions and understand if the wrong assumptions could be the culprit (e.g. the incorrect assumption that the COPY will make the current user the owner of the copied files).

Add additional debug info to Dockerfile

Sometimes it is helpful to print additional debug information when building a docker image. Some commands I used were:

• RUN ls -al
• RUN pwd
• RUN whoami

They allowed me understand the state the container was in at a given time. One caveat is that by default docker caches intermediate steps when building containers which may result in not printing the debug information when re-building a container if no changes were made as the step was cached.

Run the failing command manually and/or inspect the container

This is the ultimate debugging strategy – manually reproduce the error and inspect the container state. One way to make it work is to comment out all the steps starting from the failing one and then build the image. Once the image is build start a container like this (replace IMAGE with the image id):

docker run -d IMAGE tail -f /dev/null

This will start the container whose state is just as it was before the failing step was executed. The command will also keep the container running which makes it possible for you to launch bash within the container (replace CONTAINER with the container id returned by the previous command):

docker exec -it CONTAINER /bin/bash

Once inside the container you can run the command that was failing (e.g. npm install). Since the container is in the same state it was when it failed to build you should be able to reproduce the failure. You can also easily check for the factors that caused the failure.

Conclusion

his post showed how to create a docker container that is not running as root and discussed a few permission issues encountered in the process. It also described a few debugging strategies that can help troubleshoot a wide range of issues – including issues related to permissions. The code for this post is available on github in my docker-permissions repo.