opendut icon indicating copy to clipboard operation
opendut copied to clipboard
verified publisher icon eclipse-opendut

Upgrade netbird to latest suitable version

Open mplicht opened this issue 3 months ago • 3 comments

Due to changes in NetBird (configurable MTU, relay server, mTLS) we should upgrade NetBird to a higher version. With that we get also rid of the management of the NetBird fork.

Tasks:

  • [ ] Upgrade NetBird-Management-API in CARL, see NetBird API documentation
  • [ ] Read changelog since v0.28.9
  • [x] Upgrade NetBird-Client-API in EDGAR
    • [x] Use configurable MTU
    • [ ] Check NetBird client API
  • [ ] Remove coturn from deployment, see also NetBird management configuration
  • [ ] Add Relay Server to Localenv deployment + remove Coturn server deployment
  • [ ] Check for configuration changes here
    • Signal has new traefik rules: netbird-wsproxy-signal
    • NetBird dashboard has new environment variable NETBIRD_TOKEN_SOURCE
    • Management has new traefik rule netbird-wsproxy-mgmt, and new environment variables that need to be checked
  • [ ] Re-read the NetBird self-hosted guide
  • [ ] Archive NetBird fork

Notes:

  • Check for breaking changes and compare infrastructure files

mplicht avatar Sep 12 '25 08:09 mplicht

Current status

The NetBird integration tests seem to run through, but cargo theo testenv edgar start does not.

In the logs, I'm seeing these errors:

EDGAR Container logs

Using PeerId: 525b369f-8abb-4b49-8046-25948936ad6c
Will connect to CARL at: https://carl.opendut.local/

Task succeeded: Write CA Certificates
Task succeeded: Check availability of needed command line programs
Task succeeded: Check CARL Reachable
Task succeeded: Copy executable to "/opt/opendut/edgar/opendut-edgar"
Task succeeded: Copy the rperf distribution
Task succeeded: Load Kernel Modules vcan, can_gw (Unchanged)
Task succeeded: NetBird - Unpack
Task succeeded: NetBird - Install Service
Task succeeded: NetBird - (Re-)Start Service
    NetBird - Connect
    Error during NetBird-Login: Request error: status: 'Some requested entity was not found', self: "couldn't add peer: setup key is invalid", metadata: {"content-type": "application/grpc"}: status: 'Some requested entity was not found', self: "couldn't add peer: setup key is invalid", metadata: {"content-type": "application/grpc"}
Task failed: NetBird - Connect
+ opendut-cleo await peer-online 525b369f-8abb-4b49-8046-25948936ad6c
2025-11-24T12:09:02.404035Z TRACE opendut_telemetry: Telemetry stack initialized without OpenTelemetry.
2025-11-24T12:09:02.404162Z DEBUG opendut_carl_api::carl: Using TLS CA certificate: /provision/pki/opendut-ca.pem
2025-11-24T12:09:02.404209Z DEBUG opendut_carl_api::carl: Using override for verified domain name of 'carl.opendut.local'.
2025-11-24T12:09:02.404586Z TRACE opendut_auth::confidential::client: OIDC configuration loaded: client_id='opendut-cleo-client', issuer_url='https://auth.opendut.local/realms/opendut/'
2025-11-24T12:09:02.426800Z DEBUG opendut_carl_api::carl: Set up endpoint for connection to CARL at 'https://carl.opendut.local:443'.
2025-11-24T12:09:02.434911Z  INFO opendut_carl_api::carl: Connected to CARL at 'https://carl.opendut.local:443'.

CARL Container Logs (in a later run)

2025-11-24T12:38:38.022376Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:generate_netbird_setup_key{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:get_netbird_group{group_name=Peer(PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>)}: opendut_vpn_netbird::client::request_handler: Network request completed.
2025-11-24T12:38:38.023531Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:generate_netbird_setup_key{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}: opendut_vpn_netbird::client::request_handler: Starting network request with timeout 10s.
2025-11-24T12:38:38.023849Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:generate_netbird_setup_key{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}: opendut_vpn_netbird::client::request_handler: Sending request POST https://netbird-api.opendut.local/api/setup-keys
2025-11-24T12:38:38.066858Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<d629fede-27c8-4270-8e73-f91ae7d31a33>}:get_netbird_group{group_name=Peer(PeerId<d629fede-27c8-4270-8e73-f91ae7d31a33>)}: opendut_vpn_netbird::client::request_handler: Got response 401 Unauthorized
2025-11-24T12:38:38.067311Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<d629fede-27c8-4270-8e73-f91ae7d31a33>}:get_netbird_group{group_name=Peer(PeerId<d629fede-27c8-4270-8e73-f91ae7d31a33>)}: opendut_vpn_netbird::client::request_handler: Network request completed.
2025-11-24T12:38:38.067946Z ERROR generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<d629fede-27c8-4270-8e73-f91ae7d31a33>}: opendut_vpn_netbird: Failed to generate vpn configuration for peer <d629fede-27c8-4270-8e73-f91ae7d31a33>, due to communication issues when trying to look up the peer's self group!
2025-11-24T12:38:38.068694Z ERROR generate_peer_setup: opendut_carl::manager::grpc::error: Error during API call: An internal error occurred while creating a PeerSetup for peer 'test-environment-cluster-peer-4' <d629fede-27c8-4270-8e73-f91ae7d31a33>:
  An error occurred while creating a vpn configuration for peer <d629fede-27c8-4270-8e73-f91ae7d31a33>:
  Could not request group 'opendut-peer-group-d629fede-27c8-4270-8e73-f91ae7d31a33':
  JSON deserialization error: error decoding response body
2025-11-24T12:38:38.527168Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:generate_netbird_setup_key{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}: opendut_vpn_netbird::client::request_handler: Got response 200 OK
2025-11-24T12:38:38.527304Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}:generate_netbird_setup_key{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}: opendut_vpn_netbird::client::request_handler: Network request completed.
2025-11-24T12:38:38.527589Z DEBUG generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<bcf75b6c-d6e1-42bd-b74e-30690bca88ab>}: opendut_vpn_netbird: Successfully generated VPN configuration for peer <bcf75b6c-d6e1-42bd-b74e-30690bca88ab>.
2025-11-24T12:38:38.527670Z  INFO generate_peer_setup:generate_peer_setup: opendut_carl::manager::peer_manager::generate_peer_setup: Successfully retrieved vpn configuration for peer <bcf75b6c-d6e1-42bd-b74e-30690bca88ab>.
2025-11-24T12:38:38.527699Z DEBUG generate_peer_setup:generate_peer_setup: opendut_carl::manager::peer_manager::generate_peer_setup: Generating OIDC client for peer 'test-environment-cluster-peer-3' <bcf75b6c-d6e1-42bd-b74e-30690bca88ab>.
2025-11-24T12:38:38.573301Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>}:generate_netbird_setup_key{peer_id=PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>}:get_netbird_group{group_name=Peer(PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>)}: opendut_vpn_netbird::client::request_handler: Got response 200 OK
2025-11-24T12:38:38.573458Z TRACE generate_peer_setup:generate_peer_setup:generate_vpn_peer_configuration{peer_id=PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>}:generate_netbird_setup_key{peer_id=PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>}:get_netbird_group{group_name=Peer(PeerId<a1db14f5-1d08-4876-adf2-ba32d99f25ff>)}: opendut_vpn_netbird::client::request_handler: Network request completed.

(I'm not sure yet, if the 401 error while getting the NetBird group is reproducible.)

mbfm avatar Nov 24 '25 16:11 mbfm

  • [ ] NetBird now supports deleting Setup-Keys. We should make use of this, rather than just revoking them.

mbfm avatar Nov 25 '25 07:11 mbfm

We're still facing issues with the client login and noticed the netbird login command is also not working (unlike netbird up, which does work), but these appear to be independent. netbird login is broken due to the CLI parsing not picking up --management-url: https://github.com/netbirdio/netbird/issues/4601

mbfm avatar Nov 28 '25 07:11 mbfm

We now call the netbird up command instead of using Protobuf messages for that. We did not figure out why the Protobuf messages would not work, but this fixes the problem and the code change is not too bad (only needed in EDGAR Setup, where calling commands is normal).

Additionally, this means we're now on a somewhat official API, which seems to have not seen (intentional) breaking changes in the last two years, so hopefully the maintenance does not increase from not having compiler errors.

mbfm avatar Dec 03 '25 09:12 mbfm