haproxy-lua-acme icon indicating copy to clipboard operation
haproxy-lua-acme copied to clipboard

500 Internal Service Error on new cert order

Open michaelmedellin opened this issue 7 years ago • 6 comments

Using an identical HA Proxy configuration and curl command to the README, I can't seem to get an order to go through and either fail or succeed. Seems to fail almost immediately with a 500 error. Would appreciate some guidance—I'm fairly new to HA Proxy so I may be missing something relatively simple.

haproxy.cfg:

global
    log /dev/log local0 debug
    nbproc 1
    daemon
    lua-load config.lua
    lua-load acme.lua

defaults
    log global
    mode http
    option httplog
    timeout connect 5s
    timeout client 10s
    timeout server 10s

listen http
    bind *:80
    http-request use-service lua.acme if { path_beg /.well-known/acme-challenge/  }

listen acme
    bind 127.0.0.1:9011
    http-request use-service lua.acme

listen acme-ca
  bind 127.0.0.1:9012
  server ca acme-v02.api.letsencrypt.org:443 ssl verify required ca-file letsencrypt-x3-ca-chain.pem

CURL:

curl -XPOST -v http://127.0.0.1:9011/acme/order -F '[email protected]' \
     -F 'domain=example.net' -F '[email protected]' \
     -F 'aliases=www.example.net,example.com,www.example.com' \
     -o example.net.pem

HA Proxy Version:

HA-Proxy version 1.8.13 2018/07/30
Copyright 2000-2018 Willy Tarreau <[email protected]>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
  OPTIONS = USE_LIBCRYPT=1 USE_CRYPT_H=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Curl Failure:

*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 9011 (#0)
> POST /acme/order HTTP/1.1
> Host: 127.0.0.1:9011
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 5453
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------47a351a7e23f1bf9
>
< HTTP/1.1 500 Internal Server Error
< content-type: text/plain; charset=utf-8
< content-length: 0
* HTTP error before end of send, stop sending
<
  0  5453    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0

HA Proxy Debug Output:

00000000:acme-ca.accept(0007)=000a from [127.0.0.1:36338] ALPN=<none>
00000000:acme-ca.clireq[000a:ffffffff]: GET /dir HTTP/1.1
00000000:acme-ca.clihdr[000a:ffffffff]: host: 127.0.0.1:9012
00000000:acme-ca.clihdr[000a:ffffffff]: accept: */*
00000000:acme-ca.clihdr[000a:ffffffff]: user-agent: haproxy-lua-http/1.0
00000000:acme-ca.clihdr[000a:ffffffff]: connection: close
00000000:acme-ca.srvrep[000a:adfd]: HTTP/1.0 400 Bad Request
00000000:acme-ca.srvhdr[000a:adfd]: Server: AkamaiGHost
00000000:acme-ca.srvhdr[000a:adfd]: Mime-Version: 1.0
00000000:acme-ca.srvhdr[000a:adfd]: Content-Type: text/html
00000000:acme-ca.srvhdr[000a:adfd]: Content-Length: 209
00000000:acme-ca.srvhdr[000a:adfd]: Expires: Sun, 19 Aug 2018 14:04:05 GMT
00000000:acme-ca.srvhdr[000a:adfd]: Date: Sun, 19 Aug 2018 14:04:05 GMT
00000000:acme-ca.srvhdr[000a:adfd]: Connection: close

michaelmedellin avatar Aug 19 '18 14:08 michaelmedellin

@michaelmedellin you need to add a set-header in the acme-ca listener such as the following: http-request set-header Host acme-v02.api.letsencrypt.org

jmagnin avatar Oct 11 '18 11:10 jmagnin

It looks like this issue is still a thing... perhaps related to ACME v2?

ingrid_1  | 0000000d:acme-ca.clireq[001a:ffffffff]: GET /acme/authz-v3/54535395 HTTP/1.1
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: host: 127.0.0.1:9012
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: accept: */*
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: user-agent: haproxy-lua-http/1.0
ingrid_1  | 0000000d:acme-ca.srvrep[001a:001b]: HTTP/1.1 405 Method Not Allowed
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: server: nginx
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: date: Wed, 06 May 2020 16:31:58 GMT
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: content-type: application/problem+json
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: content-length: 103
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: cache-control: public, max-age=0, no-cache
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
ingrid_1  | 0000000d:acme-ca.srvcls[001a:001b]
ingrid_1  | 0000000d:acme-ca.clicls[001a:001b]
ingrid_1  | 0000000d:acme-ca.closed[001a:001b]
ingrid_1  | (table) table: 0x5622f280fba0 [
ingrid_1  |     "detail": (string) "Method not allowed"
ingrid_1  |     "type": (string) "urn:ietf:params:acme:error:malformed"
ingrid_1  |     "status": (number) 405
ingrid_1  | ]

I don't think it's correct to be performing a GET on the authorization. Could someone confirm this is working with LetsEncrypt v2 staging endpoint, at least to save me trying to diagnose something which isn't broken?

richbayliss avatar May 06 '20 16:05 richbayliss

Yeah, I can confirm that the errors are due to the deprecation of GET requests in v2 and switching to POST-as-GET worked in the end. I will try and PR a fix for you guys 👍

richbayliss avatar May 07 '20 14:05 richbayliss

Hello Rich,

Yep, v1 API is depreciated, we're planning upgrade for a long time, but PR are always welcome :-)

anezirovic avatar May 07 '20 14:05 anezirovic

Can you please update the README on the front page? I just spent quite a long time setting this up and working through it only to discover it does not work on account of this issue.

mianos avatar Jun 15 '20 04:06 mianos

I had a lot of issues to renew my certificates with this tool since ~1 month. 80% of my renewal requests was aborted.

After multiple tests, here are my results :

  • POST-as-GET directive was delayed for the production environment, it's not the main issue : https://community.letsencrypt.org/t/acme-v2-scheduled-deprecation-of-unauthenticated-resource-gets/74380/5
  • The current LUA script in master branch is working in ACMEv2 production environment
  • Let's Encrypt infrastructure has a lot of issues, yesterday 1/3 of my requests were 503 errors
  • These 503 errors are breaking the request chain in LUA script, aborting a lot of certificates
  • I had an issue with my chained cert letsencrypt-x3-ca-chain.pem which was too old (it was using Let’s Encrypt Authority X3 intermediate cert which is retired)

I send a pull request today with :

  • A retry mecanism to increase reliability (5 retries per request)
  • Compliance with POST-as-GET directive

delormejonathan avatar Dec 16 '20 14:12 delormejonathan