netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Published 20 hours ago • verified publisher icon netbirdio

Fedora 35 system not seeing Debian 11.1 system and vice versa

Open harishpillay opened this issue 3 years ago • 17 comments

I have setup two systems: one Fedora 34 and the other Debian 11.1. Both were registered with the Default Keys. They have both received IP#s but the https://app.wiretrustee.com/peers only shows the Debian 11 system being online while the Fedora 35 is offline. Any tips on troubleshooting?

harishpillay avatar Nov 04 '21 10:11 harishpillay

Hi @harishpillay I'm sorry that you have an issue with running Wiretrustee on Fedora.

We haven't tested Wiretrustee on Fedora extensively. We will take care of this.

We could review your log file. It is located under /var/log/wiretrustee/client.log

braginini avatar Nov 04 '21 10:11 braginini

% head -5 /var/log/wiretrustee/client.log time="2021-11-04T17:19:11+08:00" level=info msg="starting service" time="2021-11-04T17:19:11+08:00" level=error msg="failed reading config /etc/wiretrustee/config.json open /etc/wiretrustee/config.json: no such file or directory" time="2021-11-04T17:19:12+08:00" level=error msg="failed reading config /etc/wiretrustee/config.json open /etc/wiretrustee/config.json: no such file or directory" time="2021-11-04T17:19:14+08:00" level=error msg="failed reading config /etc/wiretrustee/config.json open /etc/wiretrustee/config.json: no such file or directory" time="2021-11-04T17:19:16+08:00" level=error msg="failed reading config /etc/wiretrustee/config.json open /etc/wiretrustee/config.json: no such file or directory"

is repeatedly posted although it exists:

% ls -lat /etc/wiretrustee/config.json -rw-------. 1 root root 443 Nov 4 17:24 /etc/wiretrustee/config.json

Happy to continue to test Fedora 35 (and RHEL/CentOS systems).

harishpillay avatar Nov 04 '21 11:11 harishpillay

I have installed on a CentOS Stream 8 system and I am able to ping the Debian 11.1 (and vice versa) and able to login via ssh. But the CentOS Stream 8 can't ping the Fedora 35 system.

On the F35:

-rw-------. 1 root root unconfined_u:object_r:etc_t:s0 443 Nov  4 17:24 /etc/wiretrustee/config.json

On the CentOS Stream 8:

-rw-------. 1 root root unconfined_u:object_r:etc_t:s0 443 Nov  4 20:34 /etc/wiretrustee/config.json

On the Debian 11.1:

-rw------- 1 root root ? 443 Nov  4 18:21 /etc/wiretrustee/config.json

So, SELinux does not seem to be an issue.

I will test further.

Harish

harishpillay avatar Nov 04 '21 12:11 harishpillay

thank you @harishpillay we have planned some tests on Fedora for tomorrow

braginini avatar Nov 05 '21 17:11 braginini

thanks @braginini. looking forward to the updated rpms.

harishpillay avatar Nov 06 '21 07:11 harishpillay

I've updated all the systems with 0.2.3 and only the CentOS Stream 8 and Debian 11.1 can ping each other. The F35 systems can't.

harishpillay avatar Nov 07 '21 02:11 harishpillay

Hello @harishpillay, after testing the installation, I couldn't reproduce the issue on a fresh Fedora 35 installation and Wiretrustee v0.2.3.

Checking SELinux contexts your config file seems ok, please ensure that the folder is following the same context as below:

 ls -latZ /etc/wiretrustee
total 4
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0     4648 Nov  7 10:48 ..
drwxr-x---. 1 root root unconfined_u:object_r:etc_t:s0   22 Nov  7 10:47 .
-rw-------. 1 root root unconfined_u:object_r:etc_t:s0  443 Nov  7 10:47 config.json

Can you confirm that you are running the service with sudo?

Besides that, we can check if there are any entries in the audit.log file that needs to be applied (not expected in default SELinux mode):

sudo grep wiretrustee /var/log/audit/audit.log | audit2allow

mlsmaycon avatar Nov 07 '21 10:11 mlsmaycon

@mlsmaycon

Hi. Thanks for the suggestions:

Hello @harishpillay, after testing the installation, I couldn't reproduce the issue on a fresh Fedora 35 installation and Wiretrustee v0.2.3.

Checking SELinux contexts your config file seems ok, please ensure that the folder is following the same context as below:

 ls -latZ /etc/wiretrustee
total 4
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0     4648 Nov  7 10:48 ..
drwxr-x---. 1 root root unconfined_u:object_r:etc_t:s0   22 Nov  7 10:47 .
-rw-------. 1 root root unconfined_u:object_r:etc_t:s0  443 Nov  7 10:47 config.json

the above is the exactly same in the CentOS Stream 8 and Fedora 35 systems.

Can you confirm that you are running the service with sudo?

as in systemctl? the setup done was the same on all of the systems. what would I need to do to check?

Besides that, we can check if there are any entries in the audit.log file that needs to be applied (not expected in default SELinux mode):

sudo grep wiretrustee /var/log/audit/audit.log | audit2allow

says, "Nothing to do".

On the F35:

$ tail -f /var/log/wiretrustee/client.log 
time="2021-11-07T19:13:04+08:00" level=error msg="failed reading config /etc/wiretrustee/config.json open /etc/wiretrustee/config.json: too many open files"
time="2021-11-07T19:13:21+08:00" level=error msg="failed creating connection to Management Service context deadline exceeded"
time="2021-11-07T19:13:21+08:00" level=warning msg="rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded"
time="2021-11-07T19:13:40+08:00" level=error msg="failed to connect to the signalling server context deadline exceeded"
time="2021-11-07T19:13:40+08:00" level=error msg="error while connecting to the Signal Exchange Service signal2.wiretrustee.com:10000: context deadline exceeded"
time="2021-11-07T19:13:40+08:00" level=error msg="rpc error: code = FailedPrecondition desc = failed connecting to Signal Service : context deadline exceeded"
time="2021-11-07T19:14:06+08:00" level=error msg="failed to connect to the signalling server context deadline exceeded"
time="2021-11-07T19:14:06+08:00" level=error msg="error while connecting to the Signal Exchange Service signal2.wiretrustee.com:10000: context deadline exceeded"
time="2021-11-07T19:14:06+08:00" level=error msg="rpc error: code = FailedPrecondition desc = failed connecting to Signal Service : context deadline exceeded"
time="2021-11-07T19:14:27+08:00" level=error msg="failed to connect to the signalling server context deadline exceeded"
time="2021-11-07T19:14:27+08:00" level=error msg="error while connecting to the Signal Exchange Service signal2.wiretrustee.com:10000: context deadline exceeded"
time="2021-11-07T19:14:27+08:00" level=error msg="rpc error: code = FailedPrecondition desc = failed connecting to Signal Service : context deadline exceeded"

harishpillay avatar Nov 07 '21 11:11 harishpillay 

# wg
interface: wt0
  public key: /dDJVo3o70xPhTYi65JfZC3YO/k8FZuuNWJdEAePvjc=
  private key: (hidden)
# tail -f tail -f /var/log/wiretrustee/client.log
# systemctl start wiretrustee.service 
time="2021-11-07T19:25:04+08:00" level=info msg="starting service"
time="2021-11-07T19:25:06+08:00" level=error msg="failed configuring Wireguard interface [wt0]: read: wguser: errno=-98"
time="2021-11-07T19:25:06+08:00" level=error msg="error while starting Wiretrustee Connection Engine: read: wguser: errno=-98"
time="2021-11-07T19:25:09+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-07T19:25:09+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-07T19:25:12+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-07T19:25:12+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-07T19:25:17+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-07T19:25:17+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"

strange that it says the wt0 is busy. it was created but no IP was assigned.

harishpillay avatar Nov 07 '21 11:11 harishpillay

I've done more testing and here are the results:

Setup 1: a) NUC running CentOS Stream 8 native with wt enabled. b) Debian 11.1 VM on that NUC with wt enabled. c) Fedora 35 VM on that NUC with wt enabled. This F35 VM is freshly installed from F35 ISO. d) 5 other upgraded F35 systems on the LAN that the NUC is attached to and two of them with wt enabled.

In Setup 1, all three can ping, ssh etc between themselves. The F35s in the LAN can't ping each other or with the ones in the NUC and vice versa.

Setup 2 (remote location): a) Dell server running RHEL 6 (wt NOT enabled) b) F35 VM on Dell server with wt enabled. This is a freshly installed F35. c) Upgraded F35 VM on the same Dell with wt enabled. d) 2nd instance of F35 VM on the Dell with wt enabled. This is also freshly installed.

In Setup 2, Upgraded F35 can't ping the two freshly installed F35 VMs. Those two F35 VMs can ping each other.

None of the systems from Setup 1 can access Setup 2 and vice versa.

Hope these additional scenarios are useful for debugging.

harishpillay avatar Nov 08 '21 09:11 harishpillay

hello @harishpillay could you run the following commands to enable a more verbose log:

sudo wiretrustee service stop
sudo wiretrustee service uninstall
sudo wiretrustee service install --log-level debug
sudo wiretrustee service start

After that it will be great if you can share the log files of at least 2 peers that can't communicate.

mlsmaycon avatar Nov 08 '21 23:11 mlsmaycon

OK. Here's output from two F35 systems (that were upgraded from F34):

System 1:

tail -f /var/log/wiretrustee/client.log
time="2021-11-09T10:18:59+08:00" level=debug msg="configuring Wireguard interface wt0"
time="2021-11-09T10:18:59+08:00" level=debug msg="adding Wireguard private key"
time="2021-11-09T10:18:59+08:00" level=debug msg="got Wireguard device wt0"
time="2021-11-09T10:18:59+08:00" level=error msg="failed configuring Wireguard interface [wt0]: read: wguser: errno=-98"
time="2021-11-09T10:18:59+08:00" level=error msg="error while starting Wiretrustee Connection Engine: read: wguser: errno=-98"
time="2021-11-09T10:19:01+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:01+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:01+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:02+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:02+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:04+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:05+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:05+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:06+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:06+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:09+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:09+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:10+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:10+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:10+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:13+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:14+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"

System 2:

# tail -f /var/log/wiretrustee/client.log 
time="2021-11-09T10:19:16+08:00" level=debug msg="configuring Wireguard interface wt0"
time="2021-11-09T10:19:16+08:00" level=debug msg="adding Wireguard private key"
time="2021-11-09T10:19:16+08:00" level=debug msg="got Wireguard device wt0"
time="2021-11-09T10:19:16+08:00" level=error msg="failed configuring Wireguard interface [wt0]: read: wguser: errno=-98"
time="2021-11-09T10:19:16+08:00" level=error msg="error while starting Wiretrustee Connection Engine: read: wguser: errno=-98"
time="2021-11-09T10:19:17+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:17+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:18+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:19+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:19+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:21+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:22+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:22+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:23+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:23+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:26+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:26+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:27+08:00" level=debug msg="peer logged in to Management Service api.wiretrustee.com:33073"
time="2021-11-09T10:19:28+08:00" level=error msg="failed creating interface wt0: [device or resource busy]"
time="2021-11-09T10:19:28+08:00" level=error msg="error while starting Wiretrustee Connection Engine: device or resource busy"
time="2021-11-09T10:19:31+08:00" level=debug msg="connecting to management server api.wiretrustee.com:33073"
time="2021-11-09T10:19:32+08:00" level=debug msg="connected to management server api.wiretrustee.com:33073"

Hope this helps.

harishpillay avatar Nov 09 '21 02:11 harishpillay

Thank you @harishpillay, it seems like that at some point the interface wasn't removed correctly. Can you the commands below on both systems:

sudo ip link delete wt0
sudo wiretrustee service stop
sudo wiretrustee service start

After that you can try to ping each and if needed, send us the logs for troubleshoot

mlsmaycon avatar Nov 09 '21 06:11 mlsmaycon

Did just that and no luck. Same messages as in the logs above.

Not able to ping any of the others.

harishpillay avatar Nov 09 '21 07:11 harishpillay

Can you disable SELinux and check if the interface is created ?

ip a
sudo ip link delete wt0
sudo wiretrustee service stop
sudo setenforce 0
sudo wiretrustee service start
ip a

Can you confirm if both runs of the ip a returns the wt0 interface?

Once you are done with the command, you can enable SElinux by running:

sudo setenforce 1

mlsmaycon avatar Nov 09 '21 08:11 mlsmaycon 

# ip a s wt0
22: wt0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.64.0.6/24 brd 100.64.0.255 scope global wt0
       valid_lft forever preferred_lft forever
    inet6 fe80::c955:3d5e:bf0b:41b2/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
# wiretrustee service stop
Wiretrustee service has been stopped
# setenforce 0
# wiretrustee service start
Wiretrustee service has been started
# ip a s wt0
23: wt0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.64.0.6/24 brd 100.64.0.255 scope global wt0
       valid_lft forever preferred_lft forever
    inet6 fe80::9190:f0f6:42b3:c60b/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

still not pinging and the log has the same error messages.

harishpillay avatar Nov 09 '21 08:11 harishpillay

@harishpillay my apologies for the delay in responding.

It seems like you missed the sudo ip link delete wt0 command, can you test if that works as the interface may have been created by another process.

Please run ip a s wt0 after that too.

mlsmaycon avatar Nov 10 '21 21:11 mlsmaycon