LetsEncrypt ACME redirect issue fixes #2881

[EDIflyer]

Evolution of #2038 to escape regex sequence (as per https://github.com/NginxProxyManager/nginx-proxy-manager/pull/2038#issuecomment-1372833078) and rebased against latest develop branch.

[xipox]

i don't know why NPM isn't starting when it's trying to migrate the DBs `

app_1 | - /data/nginx/stream/17.conf app_1 | - /data/nginx/stream/13.conf app_1 | - /data/nginx/stream/7.conf app_1 | - /data/nginx/stream/2.conf app_1 | - /data/nginx/stream/3.conf app_1 | ❯ Docker secrets ... app_1 | app_1 | ------------------------------------- app_1 | _ _ ____ __ __ app_1 | | \ | | _ | / | app_1 | | | | |) | |/| | app_1 | | |\ | __/| | | | app_1 | || _|| || |_| app_1 | ------------------------------------- app_1 | User: npm PUID:0 ID:0 GROUP:0 app_1 | Group: npm PGID:0 ID:0 app_1 | ------------------------------------- app_1 | app_1 | ❯ Starting nginx ... app_1 | ❯ Starting backend ... app_1 | [8/14/2023] [11:52:05 PM] [Global ] › ℹ info Using Sqlite: /data/database.sqlite app_1 | [8/14/2023] [11:52:06 PM] [Migrate ] › ℹ info Current database version: none app_1 | [8/14/2023] [11:58:34 PM] [Global ] › ✖ error Command failed: . /opt/certbot/bin/activate && pip install --no-cache-dir certbot-dns-cloudflare==$(certbot --version | grep -Eo '0-9+') cloudflare && deactivate app_1 | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe76f73b240>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/certbot-dns-cloudflare/ app_1 | WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe76f73b518>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/certbot-dns-cloudflare/ app_1 | WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe76f73b6a0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/certbot-dns-cloudflare/ app_1 | WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe76f73b7f0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/certbot-dns-cloudflare/ app_1 | WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fe76f73b940>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/certbot-dns-cloudflare/ app_1 | ERROR: Could not find a version that satisfies the requirement certbot-dns-cloudflare==2.5.0 (from versions: none) app_1 | ERROR: No matching distribution found for certbot-dns-cloudflare==2.5.0 app_1 | app_1 | [8/14/2023] [11:58:35 PM] [Migrate ] › ℹ info Current database version: none

`

[EDIflyer]

@jc21 any chance of merging this in? 🙂

[JDENredden]

This fixed my installation. Please merge.

[EDIflyer]

Thanks for approving @etabarestx 🙂 Do you know if a new release is upcoming with this and other PRs included?

[KaeTuuN]

Manually applied this fix via CLI and can confirm it works!

If anyone else doesn't want to wait for the merge, here is how to do it manually:

1. Open a shell as root in the docker container (varies, depending on your setup)
2. cd /etc/nginx/conf.d/include/ <- less typing
3. mv force-ssl.conf force-ssl.conf.bak <- don't change anything without a backup!
4. Copy the following codeblock:

cat <<'EOF' >force-ssl.conf     
# Since force-ssl.conf has now moved to the server section it overrides
# the LetsEncrypt config which is inside a location section
# Set FORCE variable in first two if tests pass and action in the third
set $FORCE "";
if ($scheme = "http") {
        set $FORCE 'H';
}
if ($request_uri !~ "^\/.well-known\/acme-challenge\/(.*)") {
        set $FORCE "${FORCE}D";
}
# If we are http and outside the LetsEncrypt directories redirect to https via 301
if ($FORCE = HD) {
        return 301 https://$host$request_uri;
}

5. Close your shell.
6. Restart the container.
7. REMINDER: This change will get lost if you update your Docker Image!

[evansharp]

Bumping to keep this alive.

[smibrandon]

I found a fix for my issue: allocating more storage space.

Running NPM in a Proxmox CT (no docker at all), and happened to catch that it was at 96% of its storage. I gave it some extra, and boom. Worked!

[Blogshot]

This has been an ongoing issue since 2022. How is this not merged yet?

As it stands, NPM will block renewals. I don't want to compromise security (-> disable "Force SSL") to enable renewals.

[tristanXme]

/bump please merge this!

[Guiorgy]

Applied this manually, but couldn't validate if it worked, since too many failed authorizations recently 😅. Will update the comment later. Edit: Didn't work for me, looks like I have a different problem.

[TailoredITRob]

Manually applied this fix via CLI and can confirm it works!

If anyone else doesn't want to wait for the merge, here is how to do it manually:

1. Open a shell as root in the docker container (varies, depending on your setup)
2. cd /etc/nginx/conf.d/include/ <- less typing
3. mv force-ssl.conf force-ssl.conf.bak <- don't change anything without a backup!
4. Copy the following codeblock:

cat <<'EOF' >force-ssl.conf     
# Since force-ssl.conf has now moved to the server section it overrides
# the LetsEncrypt config which is inside a location section
# Set FORCE variable in first two if tests pass and action in the third
set $FORCE "";
if ($scheme = "http") {
        set $FORCE 'H';
}
if ($request_uri !~ "^\/.well-known\/acme-challenge\/(.*)") {
        set $FORCE "${FORCE}D";
}
# If we are http and outside the LetsEncrypt directories redirect to https via 301
if ($FORCE = HD) {
        return 301 https://$host$request_uri;
}

5. Close your shell.
6. Restart the container.
7. REMINDER: This change will get lost if you update your Docker Image!

Until this is committed, why not just create a force-ssl.conf file on your host machine. Then add it as a volume to the container. E.g...

force-ssl.conf (in the same directory as docker-compose.yml)

# Since force-ssl.conf has now moved to the server section it overrides
# the LetsEncrypt config which is inside a location section
# Set FORCE variable in first two if tests pass and action in the third
set $FORCE "";
if ($scheme = "http") {
        set $FORCE 'H';
}
if ($request_uri !~ "^\/.well-known\/acme-challenge\/(.*)") {
        set $FORCE "${FORCE}D";
}
# If we are http and outside the LetsEncrypt directories redirect to https via 301
if ($FORCE = HD) {
        return 301 https://$host$request_uri;
}

docker-compose.yml

services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      # These ports are in format <host-port>:<container-port>
      - '80:80' # Public HTTP Port
      - '443:443' # Public HTTPS Port
      - '81:81' # Admin Web Port
      # Add any other Stream port you want to expose
      # - '21:21' # FTP

    # Uncomment the next line if you uncomment anything in the section
    environment:
      # Uncomment this if you want to change the location of 
      # the SQLite DB file within the container
      # DB_SQLITE_FILE: "/data/database.sqlite"

      # Uncomment this if IPv6 is not enabled on your host
      DISABLE_IPV6: 'true'

    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - ./force-ssl.conf:/etc/nginx/conf.d/include/force-ssl.conf   # <~~~~~~~~~~~~~~~

This way if you update your container's base image, the force-ssl.conf change will persist.

[EDIflyer]

@jc21 @Nephiel I see some releases are happening again - any chance of merging this fix for the ACME redirect issue into the latest version so I can move off v2.11.2 onto it? (I saw there were some CVEs fixed)

[Crazor]

Can confirm that this is still an ongoing issue. I was really surprised that there is such an easy way to misconfigure NPM from the GUI such that SSL renewals just stop working. Please merge!

[EDIflyer]

@jc21 any chance of getting this merged in? Unfortunately the docker image of this PR seems to have been removed. Have created PR #4147 to update the rest of the code to the latest dev branch (and hope it generates a new docker image too!)

[Guiorgy]

@EDIflyer I'm assuming that fixing merge conflicts in the current PR would've generated a docker image, or at least that's what I'd hope. Have you tried doing that before creating a duplicate PR?

[EDIflyer]

@Guiorgy no I hadn't tried that as wasn't quite sure how to do so within Github. The new one built fine - will see if I can do anything to edit this current branch and whether it re-triggers anything.

[EDIflyer]

Hmm OK I've now click on my patch and updated the fork - seems to have triggered a build so if that works I'll close the other PR. Thanks for the suggestion - as it was 'stuck' I just didn't know how to get a new one built! Also I see it is now under a different location (nginxproxymanager/nginx-proxy-manager-dev:pr-XXXX rather than the old jc21/nginx-proxy-manager:github-pr-XXXX format which explains why Watchtower was giving errors!)

[EDIflyer]

@Guiorgy OK new docker container built OK so have closed the other PR - thanks for the suggestion :)

[EDIflyer]

@jc21 I see some other PRs have recently been merged in, any chance of accepting this one? I think it would close off quite a few issues.

[PatVax]

I had the same issue with renewing while "force ssl" is enabled. I switched to docker image nginxproxymanager/nginx-proxy-manager-dev:pr-3121 and I can confirm that this PR solves the issue at least in my case. I hope it will be merged soon.

[mgkeeley]

@jc21 Adding my voice to request that this PR is merged in.

[EDIflyer]

Updated PR to latest version of dev branch. Please remember to backup any databases before updating!

[EDIflyer]

Have updated from 2.12.2 to 2.12.3 - updated OK and able to renew cert OK with SSL still active.

[HarryVasanth]

Have updated from 2.12.2 to 2.12.3 - updated OK and able to renew cert OK with SSL still active.

Can confirm it's working as intended again. 👍

[Crazor]

I've updated from 2.12.3 to nginxproxymanager/nginx-proxy-manager-dev:pr-3121 and my certs renewed just fine, even with Force SSL enabled. Can we get this merged?

[EDIflyer]

I have synced my fork to the latest development version. docker/rootfs/etc/nginx/conf.d/include/force-ssl.conf has not been changed in the latest dev version so IME this fix is still required

[nginxproxymanagerci[bot]]

Docker Image for build 12 is available on DockerHub:

nginxproxymanager/nginx-proxy-manager-dev:pr-3121

[!NOTE] Ensure you backup your NPM instance before testing this image! Especially if there are database changes. This is a different docker image namespace than the official image.

[!WARNING] Changes and additions to DNS Providers require verification by at least 2 members of the community!