openvpn icon indicating copy to clipboard operation
openvpn copied to clipboard
verified publisher icon OpenVPN

Misleading DCO error message

Open alexvelkov1 opened this issue 4 months ago • 9 comments

Hello everybody,

I have an incompatibility issue that leads to no traffic flowing through a successfully established tunnel.

I connect an OpenVPN client version 2.6.14 with DCO support and using the ovpn-dco kernel plugin version from 01.08.2025 cross-compiled for ARM with kernel version 5.15 to a Ubuntu 20.04 server which is running an older version of OpenVPN v2.4.7. I use AES-128-GCM which is available on both ends.

The client decides that it is DCO capable, the tunnel is successfully established but no traffic is possible! The warning says I should update my server to at least v2.4.5, I have even v2.4.7 which is newer, however this does not help.

You can find more detailed logs below.

Thanks

**Server info:**

# openvpn --version
OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc 
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_maintainer_mode=no enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

# uname -a
Linux test 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

**Connection logs:**

Mon Aug 25 11:55:13 2025 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Mon Aug 25 11:55:13 2025 WARNING: file '/opt/mastertest/certificates/server/server.key' is group or others accessible
Mon Aug 25 11:55:13 2025 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
Mon Aug 25 11:55:13 2025 library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Mon Aug 25 11:55:13 2025 Diffie-Hellman initialized with 2048 bit key
Mon Aug 25 11:55:13 2025 WARNING: normally if you use --mssfix and/or --fragment, you should also set --tun-mtu 1500 (currently it is 1472)
Mon Aug 25 11:55:13 2025 TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.3.1:7330
Mon Aug 25 11:55:13 2025 Socket Buffers: R=[212992->212992] S=[212992->212992]
Mon Aug 25 11:55:13 2025 UDP link local (bound): [AF_INET]10.0.3.9:7330
Mon Aug 25 11:55:13 2025 UDP link remote: [AF_INET]10.0.3.1:7330
Mon Aug 25 11:55:14 2025 TLS: Initial packet from [AF_INET]10.0.3.1:7330, sid=3edc9a60 fad784c7
Mon Aug 25 11:55:14 2025 VERIFY OK: depth=1, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=testca.company.com, [email protected]
Mon Aug 25 11:55:14 2025 VERIFY OK: depth=0, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=testdevice, [email protected]
Mon Aug 25 11:55:14 2025 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
Mon Aug 25 11:55:14 2025 peer info: IV_PROTO=746
Mon Aug 25 11:55:14 2025 WARNING: 'ifconfig' is present in local config but missing in remote config, local='ifconfig 10.5.0.2 10.5.0.1'
Mon Aug 25 11:55:14 2025 Outgoing Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Mon Aug 25 11:55:14 2025 Incoming Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Mon Aug 25 11:55:14 2025 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 1024 bit RSA
Mon Aug 25 11:55:14 2025 [testdevice] Peer Connection Initiated with [AF_INET]10.0.3.1:7330
Mon Aug 25 11:55:15 2025 TUN/TAP device tunnel1 opened
Mon Aug 25 11:55:15 2025 TUN/TAP TX queue length set to 1000
Mon Aug 25 11:55:15 2025 /sbin/ip link set dev tunnel1 up mtu 1472
Mon Aug 25 11:55:15 2025 /sbin/ip addr add dev tunnel1 local 10.5.0.2 peer 10.5.0.1
Mon Aug 25 11:55:15 2025 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Mon Aug 25 11:55:15 2025 Initialization Sequence Completed
Mon Aug 25 11:57:20 2025 TLS: new session incoming connection from [AF_INET]10.0.3.1:7330
Mon Aug 25 11:57:20 2025 VERIFY OK: depth=1, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=testca.company.com, [email protected]
Mon Aug 25 11:57:20 2025 VERIFY OK: depth=0, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=testdevice, [email protected]
Mon Aug 25 11:57:20 2025 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
Mon Aug 25 11:57:20 2025 peer info: IV_PROTO=746
Mon Aug 25 11:57:20 2025 WARNING: 'ifconfig' is present in local config but missing in remote config, local='ifconfig 10.5.0.2 10.5.0.1'
Mon Aug 25 11:57:20 2025 TLS: move_session: dest=TM_ACTIVE src=TM_UNTRUSTED reinit_src=1
Mon Aug 25 11:57:20 2025 TLS: tls_multi_process: untrusted session promoted to semi-trusted
Mon Aug 25 11:57:20 2025 Outgoing Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Mon Aug 25 11:57:20 2025 Incoming Data Channel: Cipher 'AES-128-GCM' initialized with 128 bit key
Mon Aug 25 11:57:20 2025 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 1024 bit RSA
Mon Aug 25 11:58:18 2025 event_wait : Interrupted system call (code=4)
Mon Aug 25 11:58:18 2025 Closing TUN/TAP interface
Mon Aug 25 11:58:18 2025 /sbin/ip addr del dev tunnel1 local 10.5.0.2 peer 10.5.0.1
Mon Aug 25 11:58:18 2025 SIGTERM[hard,] received, process exiting



**Client logs:**

Aug 25 10:00:32 BOX notic tunnel1 [ 1547]: OpenVPN 2.6.14 arm-linux-musleabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] [DCO]
Aug 25 10:00:32 BOX notic tunnel1 [ 1547]: library versions: OpenSSL 3.0.14 4 Jun 2024, LZO 2.10
Aug 25 10:00:32 BOX notic tunnel1 [ 1547]: DCO version: 2.0.0
Aug 25 10:00:32 BOX warn  tunnel1 [ 1551]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Aug 25 10:00:32 BOX warn  tunnel1 [ 1551]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: TCP/UDP: Preserving recently used remote address: [AF_INET]10.0.3.9:7330
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: Socket Buffers: R=[180224->180224] S=[180224->180224]
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: UDPv4 link local (bound): [AF_INET]10.0.3.1:7330
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: UDPv4 link remote: [AF_INET]10.0.3.9:7330
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: TLS: Initial packet from [AF_INET]10.0.3.9:7330, sid=ced5d2ba ae6e9675
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: VERIFY OK: depth=1, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=testca.company.com, [email protected]
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: VERIFY OK: depth=0, C=DE, ST=Bayern, L=Muenchen, O=Company, OU=IT,Test, CN=test, [email protected]
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: P2P mode NCP negotiation result: TLS_export=0, DATA_v2=0, peer-id 16777215, cipher=(not negotiated, fallback-cipher: AES-128-GCM)
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 1024 bits RSA, signature: RSA-SHA1, peer temporary key: 253 bits X25519
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: [oss30.i250] Peer Connection Initiated with [AF_INET]10.0.3.9:7330
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
Aug 25 10:00:32 BOX notic tunnel1 [ 1551]: TLS: tls_multi_process: initial untrusted session promoted to trusted
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: net_iface_new: add tunnel1 type ovpn-dco
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: DCO device tunnel1 opened
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: /sbin/openvpn-up-down.sh tunnel1 tunnel1 1472 0   init
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: Initialization Sequence Completed
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: Data Channel: cipher 'AES-128-GCM'
Aug 25 10:00:33 BOX notic tunnel1 [ 1551]: Timers: ping 10, ping-restart 120
Aug 25 10:00:34 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:00:38 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. **Upgrade your server to 2.4.5 or newer.**
Aug 25 10:00:41 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:00:46 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:00:56 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:00:56 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:01 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:11 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:22 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:26 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:32 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:41 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:51 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:01:56 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:07 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:11 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:21 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:26 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:35 BOX err   tunnel1 [ 1551]: Data Channel Offload doesn't support DATA_V1 packets. Upgrade your server to 2.4.5 or newer.
Aug 25 10:02:37 BOX notic tunnel1 [ 1551]: [oss30.i250] Inactivity timeout (--ping-restart), restarting
Aug 25 10:02:37 BOX notic tunnel1 [ 1551]: dco_del_peer: netlink reports object not found, ovpn-dco unloaded?
Aug 25 10:02:37 BOX notic tunnel1 [ 1551]: dco_del_peer: failed to send netlink message: No such file or directory (-2)
Aug 25 10:02:37 BOX notic tunnel1 [ 1551]: Closing DCO interface

alexvelkov1 avatar Aug 25 '25 13:08 alexvelkov1

The problem here is that your "OpenVPN server" is not a true "server" in the OpenVPN sense (mode server), but a peer-to-peer instance. This will not work between 2.4.x and a DCO-enabled client, because DCO requires DATA_V2 packets, which 2.4 does not do in peer-to-peer mode, and lacks the necessary support to negotiate what is needed.

The error message is, indeed, misleading, but it's hard to codify every single way people bolt together outdated software versions into correct error messages... sorry for that.

So, to fix the problem of "no packets getting through", also for the benefit of the archives

  • upgrade the server to 2.6.0 or later (which has proper NCP for P2P mode, including use of DATA_V2 packets)
  • use mode server on the server (this might need more config changes depending on your current configs, which is not included here, so hard to say)
  • use --disable-dco on the client side - which is not what you want to achieve, obviously.

cron2 avatar Aug 25 '25 15:08 cron2

Well, I think that it is the worst case for the tunnel to come UP if not fully functional, this makes it very hard to debug if the tunnel is an intermediate piece in a setup.

A few ideas how to mitigate this error case:

  • if P2P is a problem, then the test needs to be adjusted and eventually expanded
  • the best choice - handle data_v1 packets somehow, the DCO interface could work as a non-DCO one in case of incompatibilities

alexvelkov1 avatar Aug 25 '25 17:08 alexvelkov1

@alexvelkov1 P2P has already been adjusted and expanded as @cron2 said, if you have 2.6 on both sides, it wil work. There is no magic way we can change old versions.

Handling data_v1 packets with DCO is a lot of code and work for a handling a very specific corner case. Most p2p will not even have a GCM cipher configured. So that was investigated and it was decided not being worth the effort and complexity. It is almost dead code nowadays and will be very dead code in a few years but we would have to support it for a long time in the Linux kernel module.

We can probably adjust the error to say 2.6.0 when operating in p2p mode.

schwabe avatar Aug 25 '25 18:08 schwabe

OK, dropping P2P for a 2.4.x peer seems like a feasible and reasonable solution,

The question is whether DCO should ever be made available for non-AEAD algorithms, don't you think?

alexvelkov1 avatar Aug 25 '25 18:08 alexvelkov1

@alexvelkov1 yes we dropped support DCO + P2P + < 2.6.0. That is what we already did.

For next question: THe only clients that do not support AES-GCM are OpenVPN 2.3.x or older. They are ancient and unsupported for years by now. So there is no incentive to support non-AEAD algorithms in DCO.

TLS 1.3 also only supports AEAD algorithms.

schwabe avatar Aug 25 '25 19:08 schwabe

OpenVPN compatibility works better if the server (mode server or p2p with --tls-server) is newer than the client - much of the compat magic can only happen in the server, because a p2p client does not have all the necessary information to figure out whether something can work, and the p2p server has only learned in 2.6 to actually negotiate something.

What we could actually do is "give up on receipt of a DATA_V1 packet, turn off DCO, and reconnect"... not sure someone feels motivated to write and test the code, and how intrusive it is (and if it still works with the "new DCO" in 2.7 / linux 6.16+)

And of course we could check if --client is in use, and adjust the error message ("for peer-to-peer connections with DCO, the server must be 2.6 or later"). This will still misfire for some of the many different combinations of client/server configs and versions, but might be less misleading.

cron2 avatar Aug 25 '25 19:08 cron2

Since NIST and BSI currently both allow/advise using AES-128/256 in AEAD or CBC/CTR+HMAC modes, I definitely think that a "functioning" tunnel is better, even if this would mean losing performance benefits of DCO. This could be noted in the logs. The AEAD algorithms show already a better performance even if not using DCO.

So, your suggestion of reconnecting with DCO turned off, is a good one, I think. There could be plenty of setups that would benefit from this, to meet KRITIS requirements, for example.

alexvelkov1 avatar Aug 25 '25 21:08 alexvelkov1

@alexvelkov1 I am not sure what you actually want us to do. Everything you want and you think that would be better is already implemented in the current version.

We gave you enough hints to get to your goal with the old versions (switch to --server, disable dco). Reconnecting and disabling DCO is again something that probably nobody of the current developers will do since there are more important things to work on. And as @cron2 pointed out, it is not an easy feature that can have multiple headaches.

schwabe avatar Aug 25 '25 22:08 schwabe

Yes, you gave me hints, that's right.

The error message about updating my server on data_v1 packets was misleading.

I was thinking about a case where someone is connecting with a newer OpenVPN version (2.6) to an older peer (<2.6). In my opinion, a "seemless" switch from DCO to non-DCO would be more user-friendly than just demanding to update the server to v2.6 (which might not be easily done) or telling clients to explicitly disable DCO before connecting.

alexvelkov1 avatar Aug 26 '25 07:08 alexvelkov1