Companion is failing http-01 test for some web services

[cybertale]

trafficstars

Hi everyone, recently I'm planning on hosting a Nextcloud server, a Lychee server, a phpbb server and a Wordpress server, which will all be hosted with docker and let's encrypt, running behind nginx reverse proxy and get certs from this companion.

On the first day I succeeded making nextcloud server and lychee server online, with https, thanks to the Let's encrypt and this companion! But when I want to add the Wordpress server and phpbb server nextday, letsencrypt-nginx-proxy-companion refuses to generate new certs for these two containers. So I added DEBUG=true, and got the following errors(This is for wordpress, the phpbb has the same error log):

Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "blog.cybertale.cn"
  },
  "status": "invalid",
  "expires": "2019-12-24T12:05:31Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://blog.cybertale.cn/.well-known/acme-challenge/ERrH7N6Ysso0cWp-7IlpZ5DnjvdqfCPs49CoQyPjW_w [113.54.159.224]: \"\u003c!DOCTYPE html\u003e\\n\\n\u003chtml class=\\\"no-js\\\" lang=\\\"en-US\\\"\u003e\\n\\n\\t\u003chead\u003e\\n\\n\\t\\t\u003cmeta charset=\\\"UTF-8\\\"\u003e\\n\\t\\t\u003cmeta name=\\\"viewport\\\" content=\\\"width=dev\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/1778160705/75R20w",
      "token": "ERrH7N6Ysso0cWp-7IlpZ5DnjvdqfCPs49CoQyPjW_w",
      "validationRecord": [
        {
          "url": "http://blog.cybertale.cn/.well-known/acme-challenge/ERrH7N6Ysso0cWp-7IlpZ5DnjvdqfCPs49CoQyPjW_w",
          "hostname": "blog.cybertale.cn",
          "port": "80",
          "addressesResolved": [
            "113.54.159.224"
          ],
          "addressUsed": "113.54.159.224"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/1778160705/Qs2heg",
      "token": "ERrH7N6Ysso0cWp-7IlpZ5DnjvdqfCPs49CoQyPjW_w"
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/1778160705/5qqpcA",
      "token": "ERrH7N6Ysso0cWp-7IlpZ5DnjvdqfCPs49CoQyPjW_w"
    }
  ]
}

It seems like the http-01 test is failing. And in the log of the Wordpress, I got this:

52.28.236.88 - - [17/Dec/2019:12:01:41 +0000] "GET /.well-known/acme-challenge/daFHzgS9g-ACrg1LQlq73F4MDMigCufDnmGb43BE8hg HTTP/1.1" 404 21269 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

The nginx reverse proxy isn't handling the request but passed it to the Wordpress server, which should not happen, right?

I made a test, which is making a file under /var/www/html/.well-known/acme-challenge/ and access it both from the internet and in the container, turns out all ok. Here is my docker-compose.yml:

version: '2.2'

services:
  proxy:
    build: ./proxy
    restart: always
    ports:
      - 80:80
      - 443:443
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - certs:/etc/nginx/certs:ro
      - nginx:/etc/nginx/
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    depends_on:
      - lychee
      - phpbb
      - wordpress
    networks:
      - proxy-tier

  letsencrupt-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    restart: always
    volumes:
      - certs:/etc/nginx/certs
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - [email protected]
      - DEBUG=true
    networks:
      - proxy-tier
    depends_on:
      - proxy

  db:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - db:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=
    env_file:
      db.env

  nextcloud:
    image: nextcloud:apache
    container_name: nextcloud
    restart: always
    volumes:
      - nextcloud:/var/www/html
    environment:
      - MYSQL_HOST=db
      - VIRTUAL_HOST=cloud.cybertale.cn
      - LETSENCRYPT_HOST=cloud.cybertale.cn
      - [email protected]
      - NEXTCLOUD_TRUSTED_DOMAINS=cloud.cybertale.cn
    env_file:
      - db.env
    depends_on:
      - db
    networks:
      - proxy-tier
      - default

  lychee:
    image: linuxserver/lychee
    restart: always
    ports:
      - 40080:80
    environment:
      - VIRTUAL_HOST=lychee.cybertale.cn
      - LETSENCRYPT_HOST=lychee.cybertale.cn
      - [email protected]
      - PUID=1000
      - PGID=1000
    volumes:
      - /root/second_disk/lychee/config:/config
      - /root/second_disk/lychee/pictures:/pictures
    networks:
      - proxy-tier
      - default
    cpus: '2'

  mariadb-phpbb:
    image: 'bitnami/mariadb:10.3'
    environment:
      - MARIADB_USER=bn_phpbb
      - ALLOW_EMPTY_PASSWORD=yes
      - MARIADB_DATABASE=bitnami_phpbb
    volumes:
      - 'mariadb_phpbb_data:/bitnami'

  phpbb:
    build: ./phpbb
    restart: always
    expose:
      - "80"
    environment:
      - MARIADB_HOST=mariadb-phpbb
      - MARIADB_PORT_NUMBER=3306
      - PHPBB_DATABASE_USER=bn_phpbb
      - PHPBB_DATABASE_NAME=bitnami_phpbb
      - ALLOW_EMPTY_PASSWORD=yes

      - PHPBB_FIRST_NAME=Song
      - PHPBB_LAST_NAME=Qiang
      - PHPBB_FORUM_NAME=外骨骼实验室bbs
      - PHPBB_FORUM_DESCRIPTION=外骨骼实验室资源与教程分享及讨论
      - [email protected]

      - VIRTUAL_HOST=bbs.cybertale.cn
      - VIRTUAL_PORT=80
      - LETSENCRYPT_HOST=bbs.cybertale.cn
      - [email protected]
    volumes:
      - 'phpbb_data:/bitnami'
    depends_on:
      - mariadb-phpbb
    networks:
      - proxy-tier
      - default

  db-wordpress:
    image: mysql:5.7
    restart: always
    environment:
      - MYSQL_DATABASE=wordpress
      - MYSQL_USER=duke
      - MYSQL_PASSWORD=
      - MYSQL_ROOT_PASSWORD=
    volumes:
      - db-wordpress:/var/lib/mysql

  wordpress:
   image: wordpress
    expose:
      - "80"
    restart: always
    environment:
      - WORDPRESS_DB_HOST=db-wordpress
      - WORDPRESS_DB_NAME=wordpress

      - VIRTUAL_HOST=blog.cybertale.cn
      - LETSENCRYPT_HOST=blog.cybertale.cn
      - [email protected]
    depends_on:
      - db-wordpress
    volumes:
      - wordpress:/var/www/html
    networks:
      - proxy-tier
      - default

volumes:
  nextcloud:
  db:
  certs:
  vhost.d:
  nginx:
  html:
  mariadb_phpbb_data:
  phpbb_data:
  wordpress:
  db-wordpress:

networks:
  proxy-tier:

My web server is a centos 7 virtual machine running on a windows server. In general, I got 4 containers all running behind the proxy, each of them is assigned a unique subdomain name, all pointing to the same IPv4 address. But only Nextcloud and Lychee can get certs, the Wordpress and phpbb can't. Anyone have a clue on this? Any help is appreciated.

[cybertale]

Currently I'm generating certs manually with guide on this page, and it works. It seems like the problem should be in the companion, about why it's keep failing http-01 test.

[OscarKolsrud]

Hi, i can relate to this issue and have sort of an idea what happens. What it seems like is that when something is configured as a sort of "catch all" endpoint it will override the companion, so the error here is that the companion is not modifying the nginx config in some way to prevent this.

[OscarKolsrud]

Hi again, Found what i had done wrong. I used volumes for the certs and www data for nginx that was individual for each container. Simply made bound both of the volumes to the same dir and everything worked perfectly.

[qenya]

I have been having the same problem too. What worked for me was removing the AAAA DNS record (IPv6 address) for the domain. I don't understand why this has anything to do with it but it is perfectly reproducible - errors start happening again if I re-add the record.