bug: SSL error migrating to APISIX 3.9.0

[rastiehaiev]

Current Behavior

I have an APISIX Gateway 3.6.0 installation with 1 upstream, 1 service, 1 route, and 1 ssl. After upgrading to 3.9.0 the most of the requests to my API end up with SSL error like this (curl verbose response):

...
* Send failure: Broken pipe
* LibreSSL SSL_write: LibreSSL/3.3.6: error:02FFF020:system library:func(4095):Broken pipe, errno 32
* Failed sending HTTP request
* Connection #0 to host example.com left intact
curl: (55) getpeername() failed with errno 22: Invalid argument

I've managed to reproduce it locally using docker-compose. I've added steps to reproduce and created a small repo with all needed files (docker-compose.yaml, apisix config etc., ssl keys): https://github.com/rastiehaiev/apisix-migration-to-3_9_0-bug.

I tested upgrades:

1. 3.6.0 -> 3.7.0,
2. 3.7.0 -> 3.8.0,
3. 3.8.0 -> 3.9.0,
4. 3.6.0 -> 3.9.0

And the problem is migrating to 3.9.0, so it seems like some defect was introduced in 3.9.0.

Expected Behavior

API requests succeed after upgrading APISIX to 3.9.0.

Error Logs

2024/04/10 16:53:53 [alert] 56#56: *2224 ignoring stale global SSL error (SSL: error:1C800064:Provider routines::bad decrypt) while SSL handshaking, client: MY_IP, server: 0.0.0.0:9443
2024-04-10T16:53:53.994117366Z 2024/04/10 16:53:53 [crit] 56#56: *2224 SSL_do_handshake() failed (SSL: error:1C800064:Provider routines::bad decrypt) while SSL handshaking, client: MY_IP, server: 0.0.0.0:9443

Steps to Reproduce

1. Run APISIX Gateway 3.6.0 + etcd using docker-compose.
2. Create upstream, service, route and ssl like this (I used Admin API to create these entities):

---
upstreams:

  - id: api
    name: api
    desc: API upstream
    type: roundrobin
    scheme: https
    timeout:
      connect: 5
      read: 120
      send: 10
    nodes:
      - host: "httpbin.org"
        port: 443
        weight: 1

services:

  - id: api-http
    name: api-http
    plugins:
      gzip:
        min_length: 1100
        comp_level: 6
        types:
          - text/plain
          - application/json
        vary: true

routes:

  - id: api-http
    name: api-http
    uri: /*
    hosts:
      - "example.com"
    service_id: api-http
    upstream_id: api
    priority: 100

ssls:
  - id: ssl
    cert: "../certs/example.pem"
    key: "../certs/example.key"
    snis:
      - "example.com"

3. Make sure apisix installation works as expected:

for i in {1..10}; do curl -k --resolve 'example.com:9443:127.0.0.1' 'https://example.com:9443/get?query=1' -vvv -I || { echo "curl failed at attempt $i"; break; }; done

4. Run docker-compose down (and make sure etcd volume is not deleted).
5. Change the version of APISIX image from 3.6.0 to 3.9.0. Execute docker-compose up -d (there is some breaking change in configuration introduced in 3.9.0 - enable_http2 should be on apisix level - make sure to change it).
6. Try curling your APISIX gateway once again:

for i in {1..10}; do curl -k --resolve 'example.com:9443:127.0.0.1' 'https://example.com:9443/get?query=1' -vvv -I || { echo "curl failed at attempt $i"; break; }; done

Expected: requests succeed. Actual: some requests fail with SSL error.

To fix this, the entities created in step 2 should be re-created. Once they are re-created, the error disappears.

More verbose steps to reproduce are in the repository I created, please take a look: https://github.com/rastiehaiev/apisix-migration-to-3_9_0-bug.

Environment

• APISIX version (run apisix version): 3.6.0 - 3.6.9
• Operating system (run uname -a): Darwin NAME 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64 x86_64
• OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.25.3.1, built by gcc 10.2.1 20210110 (Debian 10.2.1-6), built with OpenSSL 3.2.0 23 Nov 2023, TLS SNI support enabled
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): bitnami/etcd:3.4.15
• APISIX Dashboard version, if relevant: irrelevant

[nitishfy]

I tried with version 3.6, this is what I got

[nitishfy]

Reproduced this:

[rastiehaiev]

@nitishfy Yeah, both of the responses are not expected behaviour. I think I faced both of them too.