Client ends HTTP TLS session with RST

[marcindulak]

The client ends HTTP TLS session with RST, instead of TCP FIN/FIN/ACK.

First, let me say that I'm not sure this a problem with caddy. I reproduce the problem using curl 8.9.0 and GNU Wget 1.24.5 as clients, against caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= as the server. I don't see the problem against nginx/1.27.0 as the server. The host is an M1 MacOS arm64.

To reproduce

1. Use the following Caddyfile, to start caddy container

   localhost:443 {
        tls internal {
                insecure_secrets_log /srv/SSLKEYLOGFILE
        }
        handle / {
                respond "Hello, HTTP version {http.request.proto}!"
        }
   }
   

2. Start the container

   docker run --detach --rm -v $PWD/Caddyfile:/etc/caddy/Caddyfile -v $PWD:/srv --name caddy caddy:2.8.4-alpine
   

3. Install tools into the container and start tcpdump

   docker exec -it caddy sh -c "apk add procps vim curl iproute2 tcpdump wireshark-common tshark wget"
   docker exec -it caddy sh -c "tcpdump -i lo -w /srv/test.pcap '(tcp or udp) and port 443'"
   

4. In other terminal, make HTTPS request, and afterwards stop the tcpdump with ctrl+c

   docker exec -it caddy sh -c "curl -vI --http1.1 https://localhost"
   

5. Decrypt the traffic

   docker exec -it caddy sh -c "editcap --inject-secrets tls,SSLKEYLOGFILE test.pcap test.pcapng"
   

Result

Here is the packet sequence from caddy as the server, read with docker exec -it caddy sh -c "tshark -r test.pcapng" Note that last RST packet

    1   0.000000    127.0.0.1 → 127.0.0.1    TCP 74 53212 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM TSval=2032565199 TSecr=0 WS=128
    2   0.000027    127.0.0.1 → 127.0.0.1    TCP 74 443 → 53212 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM TSval=2032565199 TSecr=2032565199 WS=128
    3   0.000036    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=2032565200 TSecr=2032565199
    4   0.003347    127.0.0.1 → 127.0.0.1    TLSv1 583 Client Hello (SNI=localhost)
    5   0.003367    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=2032565203 TSecr=2032565203
    6   0.005629    127.0.0.1 → 127.0.0.1    TLSv1.3 1495 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished, New Session Ticket
    7   0.005635    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [ACK] Seq=518 Ack=1430 Win=64256 Len=0 TSval=2032565205 TSecr=2032565205
    8   0.016725    127.0.0.1 → 127.0.0.1    TLSv1.3 130 Change Cipher Spec, Finished
    9   0.017028    127.0.0.1 → 127.0.0.1    HTTP 161 HEAD / HTTP/1.1 
   10   0.017039    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [ACK] Seq=1430 Ack=677 Win=65536 Len=0 TSval=2032565217 TSecr=2032565216
   11   0.017148    127.0.0.1 → 127.0.0.1    HTTP 252 HTTP/1.1 200 OK 
   12   0.017359    127.0.0.1 → 127.0.0.1    TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   13   0.017599    127.0.0.1 → 127.0.0.1    TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   14   0.017613    127.0.0.1 → 127.0.0.1    TCP 66 443 → 53212 [FIN, ACK] Seq=1640 Ack=701 Win=65536 Len=0 TSval=2032565217 TSecr=2032565217
   15   0.018616    127.0.0.1 → 127.0.0.1    TCP 66 53212 → 443 [RST, ACK] Seq=701 Ack=1641 Win=65536 Len=0 TSval=2032565218 TSecr=2032565217

Here is the corresponding packet sequence from nginx as the server.

    1   0.000000   172.30.0.5 → 172.30.0.3   TCP 74 45948 → 443 [SYN] Seq=0 Win=32120 Len=0 MSS=1460 SACK_PERM TSval=2534430028 TSecr=0 WS=4
    2   0.000089   172.30.0.3 → 172.30.0.5   TCP 74 443 → 45948 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=3424397642 TSecr=2534430028 WS=128
    3   0.000102   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=1 Ack=1 Win=32120 Len=0 TSval=2534430028 TSecr=3424397642
    4   0.005571   172.30.0.5 → 172.30.0.3   TLSv1 464 Client Hello (SNI=server-nginx)
    5   0.005696   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [ACK] Seq=1 Ack=399 Win=64768 Len=0 TSval=3424397647 TSecr=2534430033
    6   0.008106   172.30.0.3 → 172.30.0.5   TLSv1.3 1701 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished
    7   0.008126   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=399 Ack=1636 Win=31312 Len=0 TSval=2534430036 TSecr=3424397650
    8   0.008422   172.30.0.5 → 172.30.0.3   TLSv1.3 72 Change Cipher Spec
    9   0.009052   172.30.0.5 → 172.30.0.3   TLSv1.3 140 Finished
   10   0.009118   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [ACK] Seq=1636 Ack=479 Win=64768 Len=0 TSval=3424397651 TSecr=2534430036
   11   0.009286   172.30.0.3 → 172.30.0.5   TLSv1.3 369 New Session Ticket
   12   0.009327   172.30.0.3 → 172.30.0.5   TLSv1.3 369 New Session Ticket
   13   0.010075   172.30.0.5 → 172.30.0.3   HTTP 168 HEAD / HTTP/1.1 
   14   0.010429   172.30.0.3 → 172.30.0.5   HTTP 326 HTTP/1.1 200 OK 
   15   0.010849   172.30.0.5 → 172.30.0.3   TLSv1.3 90 Alert (Level: Warning, Description: Close Notify)
   16   0.010894   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [FIN, ACK] Seq=605 Ack=2502 Win=31320 Len=0 TSval=2534430039 TSecr=3424397652
   17   0.011040   172.30.0.3 → 172.30.0.5   TCP 66 443 → 45948 [FIN, ACK] Seq=2502 Ack=606 Win=64768 Len=0 TSval=3424397653 TSecr=2534430039
   18   0.011045   172.30.0.5 → 172.30.0.3   TCP 66 45948 → 443 [ACK] Seq=606 Ack=2503 Win=31320 Len=0 TSval=2534430039 TSecr=3424397653

The problem with ending RST is also present for HTTP2.

The pcapng files are attached.

Contain RST, curl/wget request to caddy http2.pcapng.gz http1.1.pcapng.gz http1.1wget.pcapng.gz

Without RST, curl request to nginx http1.1nginx.pcapng.gz