Repeated dead peer detection connection drops since upgrading to PanOS 11.0.3

[tomelliff]

We have multiple endpoints across the world and we recently rolled out an upgrade from 11.0.1 to 11.0.3 on one of them. After this upgrade our Linux users using this client saw regular connectivity issues where there was a multi second drop in connectivity.

Checking the logs by running the gpclient CLI command shows repeated dead peer detection error messages and a reconnect:

2023-11-17 09:27:49.125 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/hipreportcheck.esp

2023-11-17 09:28:20.264 INFO  [1954161] [GPClient::onVPNLogAvailable@518] GPST Dead Peer Detection detected dead peer!

2023-11-17 09:28:20.305 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/getconfig.esp

2023-11-17 09:28:20.312 INFO  [1954161] [GPClient::onVPNLogAvailable@518] SSL negotiation with broken-endpoint.example.com

2023-11-17 09:28:20.330 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Connected to HTTPS on broken-endpoint.example.com with ciphersuite (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA256)-(AES-256-GCM)

2023-11-17 09:28:20.342 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Tunnel timeout (rekey interval) is 180 minutes.
Idle timeout is 180 minutes.
POST https://broken-endpoint.example.com/ssl-vpn/hipreportcheck.esp

2023-11-17 09:28:53.036 INFO  [1954161] [GPClient::onVPNLogAvailable@518] GPST Dead Peer Detection detected dead peer!

2023-11-17 09:28:53.075 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/getconfig.esp

2023-11-17 09:28:53.081 INFO  [1954161] [GPClient::onVPNLogAvailable@518] SSL negotiation with broken-endpoint.example.com

2023-11-17 09:28:53.096 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Connected to HTTPS on broken-endpoint.example.com with ciphersuite (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA256)-(AES-256-GCM)

2023-11-17 09:28:53.108 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Tunnel timeout (rekey interval) is 180 minutes.
Idle timeout is 180 minutes.

2023-11-17 09:28:53.108 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/hipreportcheck.esp

2023-11-17 09:29:56.008 INFO  [1954161] [GPClient::onVPNLogAvailable@518] GPST Dead Peer Detection detected dead peer!

2023-11-17 09:29:56.047 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/getconfig.esp

2023-11-17 09:29:56.054 INFO  [1954161] [GPClient::onVPNLogAvailable@518] SSL negotiation with broken-endpoint.example.com

2023-11-17 09:29:56.072 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Connected to HTTPS on broken-endpoint.example.com with ciphersuite (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA256)-(AES-256-GCM)

2023-11-17 09:29:56.085 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Tunnel timeout (rekey interval) is 180 minutes.
Idle timeout is 180 minutes.
POST https://broken-endpoint.example.com/ssl-vpn/hipreportcheck.esp

2023-11-17 09:32:09.032 INFO  [1954161] [GPClient::onVPNLogAvailable@518] GPST Dead Peer Detection detected dead peer!

2023-11-17 09:32:09.068 INFO  [1954161] [GPClient::onVPNLogAvailable@518] POST https://broken-endpoint.example.com/ssl-vpn/getconfig.esp

2023-11-17 09:32:09.074 INFO  [1954161] [GPClient::onVPNLogAvailable@518] SSL negotiation with broken-endpoint.example.com

2023-11-17 09:32:09.087 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Connected to HTTPS on broken-endpoint.example.com with ciphersuite (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA256)-(AES-256-GCM)

2023-11-17 09:32:09.099 INFO  [1954161] [GPClient::onVPNLogAvailable@518] Tunnel timeout (rekey interval) is 180 minutes.
Idle timeout is 180 minutes.
POST https://broken-endpoint.example.com/ssl-vpn/hipreportcheck.esp

This behaviour wasn't seen before and if we connect to a PanOS 11.0.1 endpoint then we don't see this behaviour there either.

Using Palo Alto's GlobalProtect Linux client v6.1.2-82 also doesn't have this dead peer connectivity drop off when talking to the PanOS 11.0.3 endpoint and our Windows and Mac users using the Palo Alto GlobalProtect clients also haven't been affected. Unfortunately that Linux client doesn't support Yubikeys currently so isn't an option for us right now.

Is there a good way to further debug why this OpenConnect client is hitting this dead peer detection with the recent upgrade?

[zer0tec]

same behaviour, but on version 10.2.7.h3, only valid for SSL, no drop with ESP

[yuezk]

Closing it for now, reopen it if it is still a problem in 2.x, and I'm happy to help.