Move default writable paths to a dedicated directory

[dpkirchner]

Currently, if you try to run openresty in a K8S cluster that is configured for higher security[0], perhaps runAsUser and/or readOnlyRootFilesystem, you'll get startup errors when openresty attempts to create /usr/local/openresty/nginx/client_body_temp (and etc). To start, you'd need to mount a read-write partition into /usr/local/openresty/nginx, which would override everything there, including the binaries. Even if you were to copy the existing files into the RW mount, you'd open yourself up to bad actors modifying files that they shouldn't.

Ideally, all writable files would be in a dedicated directory, something like /var/run/openresty. That'd include all of the *_temp directories, pid, and possibly logs. Then we can ensure that nobody can replace /usr/local/openresty/nginx/** at runtime.

[0] https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#8-run-containers-as-a-non-root-user

[neomantra]

I'm very supportive of this. Thanks for pointing out K8S aspects of it.

We've talked about some aspects of this in #24. Things have evolved since then and Nginx now uses UID 101 (https://github.com/nginxinc/docker-nginx/blob/e3bbc1131a683dabf868268e62b9d3fbd250191b/mainline/buster/Dockerfile#L12).

Nginx images also symlink logs to /var/log/nginx (https://github.com/nginxinc/docker-nginx/blob/e3bbc1131a683dabf868268e62b9d3fbd250191b/mainline/buster/Dockerfile#L96). Maybe we should put our symlinks to /var/log/openresty. Not sure why, but I like that they put that as a distinct build step.

For the rest, I think we can put something like this in the http section of our nginx.conf:

    client_body_temp_path /var/run/openresty/nginx-client-body;
    proxy_temp_path /var/run/openresty/nginx-proxy;
    fastcgi_temp_path /var/run/openresty/nginx-fastcgi;
    uwsgi_temp_path /var/run/openresty/nginx-uwsgi;
    scgi_temp_path /var/run/openresty/nginx-scgi;

[dpkirchner]

I haven't been running this in production yet but the _temp_path parameters you've specified work as expected. I ended up sending logs to stdout, which might be a good default -- it'd follow the principle of least surprise, to a degree. I think it'd be useful to include examples of methods for configuring external logging and logging to files within the container (or a sibling), however I don't have any, yet.

What would be the next step? I don't know if I can volunteer yet -- maybe after a spec has been worked out.

[neomantra]

The logs would get written to /var/run/openresty/{access,error}.log but those are symlinked to /dev/std{out,err} in the build. Users can replace that if they want, but it gets Docker logging to work as expected out of the box.

Next step is to actually do the changes and test them (like you have just helped do). Which image flavor do you use? I can do a branch and work on that one and you can help test in your k8s environment. I don't have one handy for this purpose.

[neomantra]

I have released the new temporary paths in 1.15.8.2-2.

The log destinations and default user have not changed.

[grzesuav]

Hi, I am using 1.15.8.2-4 and have similar issue :

 mkdir() "/var/run/openresty/nginx-client-body" failed (13: Permission denied)

so additionally to provided fix I should mount tmp dir in k8s (@dpkirchner ) pointing to /var/run/openresty/ and writable by user?

[dubcdr]

@grzesuav did you figure out a solution? im having the same issue

[grzesuav]

@dubcdr AFAIR I changed ownership of directory, will check tomorrow to be exact

[grzesuav]

yeah, I needed to set :

        client_body_temp_path /var/run/openresty/nginx-client-body;
        proxy_temp_path       /var/run/openresty/nginx-proxy;
        fastcgi_temp_path     /var/run/openresty/nginx-fastcgi;
        uwsgi_temp_path       /var/run/openresty/nginx-uwsgi;
        scgi_temp_path        /var/run/openresty/nginx-scgi;

in nginx.conf in order to overcome that

[grzesuav]

and additionally (in k8s) I mounted tmp directory there:

        - name: tmp
          mountPath: /var/run/openresty/

where :

      - name: tmp
        emptyDir: {}

[puja-m-patil]

I'm getting same issue with Docker FROM openresty/openresty:bionic or any other latest versions. But its working with FROM openresty/openresty:trusty We are using Docker with nginx, can anybody please help me with this?

Error -

2023/08/14 16:27:26 [emerg] 7#7: mkdir() "/var/run/openresty/nginx-client-body" failed (13: Permission denied) 3 nginx: [emerg] mkdir() "/var/run/openresty/nginx-client-body" failed (13: Permission denied)

[neomantra]

Hello, can you try with a version-specific tag to prevent any ambiguity? The last trusty push was a long time ago (apparently 5 years). Wanna try 1.21.4.2-0-bionic ?

[puja-m-patil]

1.21.4.2-0-bionic

Thank you for quick reply. Sure @neomantra trying right away.

One more thing to add we are using Lua as well.

[puja-m-patil]

Hello, can you try with a version-specific tag to prevent any ambiguity? The last trusty push was a long time ago (apparently 5 years). Wanna try 1.21.4.2-0-bionic ?

Tried with FROM openresty/openresty:1.21.4.2-0-bionic still same @neomantra

[neomantra]

OK, How are you running it, etc?

[puja-m-patil]

OK, How are you running it, etc?

docker.txt PFA

[puja-m-patil]

I'm very supportive of this. Thanks for pointing out K8S aspects of it.

We've talked about some aspects of this in #24. Things have evolved since then and Nginx now uses UID 101 (https://github.com/nginxinc/docker-nginx/blob/e3bbc1131a683dabf868268e62b9d3fbd250191b/mainline/buster/Dockerfile#L12).

Nginx images also symlink logs to /var/log/nginx (https://github.com/nginxinc/docker-nginx/blob/e3bbc1131a683dabf868268e62b9d3fbd250191b/mainline/buster/Dockerfile#L96). Maybe we should put our symlinks to /var/log/openresty. Not sure why, but I like that they put that as a distinct build step.

For the rest, I think we can put something like this in the http section of our nginx.conf:

    client_body_temp_path /var/run/openresty/nginx-client-body;
    proxy_temp_path /var/run/openresty/nginx-proxy;
    fastcgi_temp_path /var/run/openresty/nginx-fastcgi;
    uwsgi_temp_path /var/run/openresty/nginx-uwsgi;
    scgi_temp_path /var/run/openresty/nginx-scgi;

@neomantra I tried adding above things as well in nginx.config still issue is persisting. in my case nginx.cofig having Lua as well inside http {.... init_worker_by_lua_block { .... } ......}

[puja-m-patil]

Hello, @neomantra or anyone else can help here?

[FieteO]

I also encountered this issue and (think I) fixed it by transferring ownership of the openresty related files to my non-root user user. Here's a Dockerfile that explains the issue:

FROM openresty:1.25.3.1-0-alpine

RUN ls -lh /var/run/
RUN ls -lh /usr/local/

ARG USER=user
ARG UID=1000
ARG GID=$UID
# Create the user
RUN addgroup -S $GID && adduser -S -s /bin/sh $USER -u $UID -G $GID \
  && chown $UID /etc/nginx/conf.d/default.conf

RUN chown user /var/run/openresty
RUN chown -R user /usr/local/openresty
RUN ls -lh /var/run/
RUN ls -lh /usr/local/openresty
USER user

CMD ["nginx", "-g", "daemon off;"]

Here is the build output. Pay special attention to the ls -lh commands that show ownership of /var/run/openresty and /usr/local/openresty:

$ docker build -t nonrootresty --no-cache --progress=plain .
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context:
#1 transferring context: 2B done
#1 DONE 0.3s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 520B done
#2 DONE 0.4s

#3 [internal] load metadata for openresty:1.25.3.1-0-alpine
#3 DONE 1.1s

#4 [1/8] FROM openresty:1.25.3.1-0-alpine@sha256:87c89a5fa4aa52701656481c288337d7735b7470675b4cca33c6e33057655c30
#4 CACHED

#5 [2/8] RUN ls -lh /var/run/
#5 0.884 total 4K     
#5 0.885 drwxr-xr-x    2 root     root        4.0K Jan 10 03:32 openresty
#5 DONE 1.4s

#6 [3/8] RUN ls -lh /usr/local/
#6 1.226 total 16K    
#6 1.226 drwxr-xr-x    2 root     root        4.0K Dec  7 09:43 bin
#6 1.226 drwxr-xr-x    1 root     root        4.0K Jan 10 03:32 lib
#6 1.226 drwxr-xr-x    1 root     root        4.0K Jan 10 03:32 openresty
#6 1.226 drwxr-xr-x    1 root     root        4.0K Jan 10 03:32 share
#6 DONE 2.2s

#7 [4/8] RUN addgroup -S 1000 && adduser -S -s /bin/sh user -u 1000 -G 1000   && chown 1000 /etc/nginx/conf.d/default.conf
#7 DONE 2.7s

#8 [5/8] RUN chown user /var/run/openresty
#8 DONE 2.0s

#9 [6/8] RUN chown -R user /usr/local/openresty
#9 DONE 47.2s

#10 [7/8] RUN ls -lh /var/run/
#10 1.278 total 4K     
#10 1.278 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 openresty
#10 DONE 2.0s

#11 [8/8] RUN ls -lh /usr/local/openresty
#11 1.424 total 324K   
#11 1.424 -rw-r--r--    1 user     root       22.4K Jan 10 03:32 COPYRIGHT
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 bin
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 luajit
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 lualib
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 nginx
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:27 openssl
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:28 pcre
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 pod
#11 1.424 -rw-r--r--    1 user     root      235.0K Jan 10 03:32 resty.index
#11 1.424 drwxr-xr-x    1 user     root        4.0K Jan 10 03:32 site
#11 DONE 1.9s

#12 exporting to image
#12 exporting layers
#12 exporting layers 3.6s done
#12 writing image sha256:cf695052d7571efc3507848673b832aef336d3464a7113ba3c9d5031db4164d4
#12 writing image sha256:cf695052d7571efc3507848673b832aef336d3464a7113ba3c9d5031db4164d4 0.1s done
#12 naming to docker.io/library/nonrootresty 0.1s done
#12 DONE 3.7s

If there is a better solution, or I am doing something wrong, I'd appreciate comments.