illumos-joyent
illumos-joyent copied to clipboard
TritonDataCenter
Network Utilities not behaving properly in LX
Seems the following commands are behaving strangely on LX. nmap nslookup ping iperf3
I decided to test them all on the following lx branded zone images.
Platform:
[root@smartos01 /zones/smith]# uname -a
SunOS smartos01 5.11 joyent_20180329T002644Z i86pc i386 i86pc
Images:
[root@smartos01 /zones/smith]# imgadm list
UUID NAME VERSION OS TYPE PUB
19aa3328-0025-11e7-a19a-c39077bfd4cf alpine-3 20170303 linux lx-dataset 2017-03-03
7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b ubuntu-16.04 20170403 linux lx-dataset 2017-04-03
3dbbdcca-2eab-11e8-b925-23bf77789921 centos-7 20180323 linux lx-dataset 2018-03-23
d3a765d6-381c-11e8-a2f0-8b4f8f1f2cca debian-9 20180404 linux lx-dataset 2018-04-04
VM's running those images:
[root@smartos01 /zones/smith]# vmadm list
UUID TYPE RAM STATE ALIAS
83cdfbec-deb1-c2a9-b717-88d09f053328 LX 2048 running alpine
b38a0cc7-83ba-636e-9dc2-c526ba6569f6 LX 2048 running centos
c149701e-2df6-699c-ee55-bc690b53211f LX 2048 running debian
e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa LX 2048 running ubuntu
The Debian Behavior:
Debian image : d3a765d6-381c-11e8-a2f0-8b4f8f1f2cca
root@c149701e-2df6-699c-ee55-bc690b53211f:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@c149701e-2df6-699c-ee55-bc690b53211f:~# nslookup google.com
../../../../lib/isc/unix/socket.c:2881: setsockopt(20, IP_RECVTOS) failed: Protocol not available
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: google.com
Address: 172.217.13.238
root@c149701e-2df6-699c-ee55-bc690b53211f:~# nmap -p 3000 google.com -Pn
Starting Nmap 7.40 ( https://nmap.org ) at 2018-04-05 19:03 UTC
dnet: Failed to open device eth0
QUITTING!
root@c149701e-2df6-699c-ee55-bc690b53211f:~# ping google.com
connect: Network is unreachable
root@c149701e-2df6-699c-ee55-bc690b53211f:~# ping -4 google.com
ping: socket: Protocol not supported
root@c149701e-2df6-699c-ee55-bc690b53211f:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.227.202, port 65351
[ 5] local 192.168.227.200 port 5201 connected to 192.168.227.202 port 53935
iperf3: getsockopt - Protocol not available
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 581 MBytes 4.87 Gbits/sec
iperf3: getsockopt - Protocol not available
[ 5] 1.00-2.00 sec 508 MBytes 4.26 Gbits/sec
root@c149701e-2df6-699c-ee55-bc690b53211f:~# iperf3 -c 192.168.227.202
Connecting to host 192.168.227.202, port 5201
[ 4] local 192.168.227.200 port 53745 connected to 192.168.227.202 port 5201
iperf3: getsockopt - Protocol not available
The Ubuntu Behavior:
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# nslookup google.com
../../../../lib/isc/unix/socket.c:2881: setsockopt(20, IP_RECVTOS) failed: Protocol not available
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: google.com
Address: 172.217.15.78
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# nmap -p 3000 google.com -Pn
Starting Nmap 7.01 ( https://nmap.org ) at 2018-04-05 19:32 UTC
dnet: Failed to open device eth0
QUITTING!
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# ping google.com
PING google.com (172.217.7.174) 56(84) bytes of data.
64 bytes from iad30s09-in-f174.1e100.net (172.217.7.174): icmp_seq=1 ttl=128 time=9.75 ms
64 bytes from iad30s09-in-f174.1e100.net (172.217.7.174): icmp_seq=2 ttl=128 time=8.83 ms
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# iperf3 -c 192.168.227.200
Connecting to host 192.168.227.200, port 5201
[ 4] local 192.168.227.202 port 64358 connected to 192.168.227.200 port 5201
^Ciperf3: getsockopt - Protocol not available
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-0.56 sec 348 MBytes 5.24 Gbits/sec 0 0.00 Bytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-0.56 sec 348 MBytes 5.24 Gbits/sec 0 sender
[ 4] 0.00-0.56 sec 0.00 Bytes 0.00 bits/sec receiver
iperf3: interrupt - the client has terminated
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
root@e655f498-8aa1-6fad-e5b0-9bd3b54fcdaa:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.227.200, port 34870
[ 5] local 192.168.227.202 port 5201 connected to 192.168.227.200 port 53745
iperf3: getsockopt - Protocol not available
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 644 MBytes 5.41 Gbits/sec
The CentOS Behavior
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# nslookup google.com
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
Name: google.com
Address: 172.217.13.238
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# ping google.com
PING google.com (172.217.7.174) 56(84) bytes of data.
64 bytes from iad30s09-in-f14.1e100.net (172.217.7.174): icmp_seq=1 ttl=128 time=10.2 ms
^C
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 10.217/10.217/10.217/0.000 ms
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# nmap -p 3000 google.com -Pn
Starting Nmap 6.40 ( http://nmap.org ) at 2018-04-05 19:39 UTC
dnet: Failed to open device eth0
QUITTING!
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.227.202, port 42752
iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host
iperf3: error - unable to start listener for connections: Address already in use
iperf3: exiting
[root@b38a0cc7-83ba-636e-9dc2-c526ba6569f6 ~]# iperf3 -c 192.168.227.202
Connecting to host 192.168.227.202, port 5201
iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host
The Alpine Behavior
83cdfbec-deb1-c2a9-b717-88d09f053328:~# cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.5.2
PRETTY_NAME="Alpine Linux v3.5"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"
83cdfbec-deb1-c2a9-b717-88d09f053328:~# nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve
Name: google.com
Address 1: 172.217.13.238 iad23s61-in-f14.1e100.net
Address 2: 2607:f8b0:4004:810::200e iad23s63-in-x0e.1e100.net
83cdfbec-deb1-c2a9-b717-88d09f053328:~# ping google.com
PING google.com (172.217.7.174): 56 data bytes
64 bytes from 172.217.7.174: seq=0 ttl=128 time=8.527 ms
64 bytes from 172.217.7.174: seq=1 ttl=128 time=9.087 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 8.527/8.807/9.087 ms
83cdfbec-deb1-c2a9-b717-88d09f053328:~# nmap -p 3000 google.com -Pn
Starting Nmap 7.40 ( https://nmap.org ) at 2018-04-05 19:42 UTC
dnet: Failed to open device eth0
QUITTING!
83cdfbec-deb1-c2a9-b717-88d09f053328:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.227.202, port 44933
[ 5] local 192.168.227.205 port 5201 connected to 192.168.227.202 port 34527
iperf3: getsockopt - Protocol not available
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 275 MBytes 2.31 Gbits/sec
iperf3: getsockopt - Protocol not available
[ 5] 1.00-2.00 sec 248 MBytes 2.08 Gbits/sec
[ 5] 1.00-2.00 sec 248 MBytes 2.08 Gbits/sec
83cdfbec-deb1-c2a9-b717-88d09f053328:~# iperf3 -c 192.168.227.202
Connecting to host 192.168.227.202, port 5201
[ 4] local 192.168.227.205 port 58287 connected to 192.168.227.202 port 5201
iperf3: getsockopt - Protocol not available
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 289 MBytes 2.42 Gbits/sec 32767 -65011712.00 Bytes
iperf3: getsockopt - Protocol not available
@Teutone I saw there is a PR opened on August 16, is there anything stopping it from getting merged?
I don't think so - just normal merge process, I don't know how long it takes them. I tried the changes in lx-brand instances (self-build ping from iputils with mentioned fix) and there were no issues. Working as expected.
I'm also seeing this kind of error.
socket.c setsockopt(20, IP_RECVTOS) failed: Protocol not available
There are various issues here. E.g., we have no support for the IP_RECVTOS option in the native kernel, but adding support for it probably wouldn't be too bad. As for the other issues: I've have to fire up some lx zones and look more closely. In any event I'll assign this to myself and see if I can't at least file proper issues away and maybe even fix some of them.
Quick Update,
iperf3 has stopped working for me in CentOS LX.
[root@7abb8f3f-697d-4846-f348-c88cf605257f ~]# iperf3 -c xx.xx.xx.xx Connecting to host xx.xx.xx.xx, port 5201 iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host
On paltform SunOS ac-1f-6b-a5-af-5a 5.11 joyent_20190718T005708Z i86pc i386 i86pc
Using this image: 3dbbdcca-2eab-11e8-b925-23bf77789921
@Smithx10 - did you recompile iperf3 or update it from pkgsrc/other-upstream? I've seen (mis)behavior like this when a program assumes existence of X implies existence of Y. In your case Y == TCP_CONGESTION, but it's not at all clear what X is.
@danmcd
I used
yum install -y iperf3
In centos lx. Image 3dbbdcca-2eab-11e8-b925-23bf77789921.
Okay, so TCP_CONGESTION did show up v. recently. I'm curious about what parameter was passed for the LX socket option? Maybe Linux does that differently than us (we use a "struct cc_algo") and we have to account for that?
I just integrated OS-7427 which adds support for TCP_CONGESTION and several other congestion control interfaces, so iperf should be able to run on the release later this week. (There are still some issues with getsockopt(TCP_INFO); OS-4525 covers that.)
I receive the following error as well when attempting to start the bind service on LX CentOS 7 (3dbbdcca-2eab-11e8-b925-23bf77789921):
setsockopt(519, IP_RECVTOS) failed: Protocol not available
I receive the following error as well when attempting to start the
bindservice on LX CentOS 7 (3dbbdcca-2eab-11e8-b925-23bf77789921):setsockopt(519, IP_RECVTOS) failed: Protocol not available
Sounds like a new issue. Please file and link here?
Sounds like a new issue. Please file and link here?
Sure. https://github.com/joyent/illumos-joyent/issues/330
This is also happening to php-fpm at line https://github.com/php/php-src/blob/df4c27642efb2ee986ad7fb744f76ab380cf5a4c/sapi/fpm/fpm/fpm_sockets.c#L501
https://www.illumos.org/issues/14744 tracks this. No guarantees on whether or not this makes it upstream. Once it does, we'll need to plumb it straight into LX. Anyone who's interested in tackling this should try and get it ready for -gate first, but if you wish to put it straight into illumos-joyent, I expect LX plumbing to arrive with it.
@Adel-Magebinary FWIW, php-fpm works well under SmartOS LX if you configure it to communicate with the web server via a Unix domain socket (i.e. a filesystem socket, rather than an inet socket).
In the php-fpm config:
[somePoolName]
listen = /var/run/php-fpm/somePoolName
listen.backlog = 256
…
In the httpd config:
<VirtualHost …>
<Directory …>
SetHandler proxy:unix:/var/run/php-fpm/somePoolName|fcgi://localhost/
…
@smokris, thanks for that. For some single-used / locally used fpm services, we used the socket instead of TCP. However, in our case, we need fpm to be available for other nodes.
eg
upstream m2_fastcgi_backend {
ip_hash
server unix:/var/run/php-fpm.sock;
server fpm-01.cns backup resolve;
server fpm-02.cns backup resolve;
server fpm-03.cns backup resolve;
}
whereas fpm-01.cns needs to listen to TCP.
This also happens in MySQL clusters.
2022-06-17T04:08:08.424303Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.424355Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.424388Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.657658Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.659807Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.659869Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supported 2022-06-17T04:08:08.660141Z 0 [Note] [MY-000000] [WSREP-SST] Cannot open netlink socket: Protocol not supporte
@Adel-Magebinary netlink sockets are a whole other world, and LX isn't necessarily designed to keep up there.
I wonder if MySQL (or mariadb) via pkgsrc on a native zone might suit you better? It is available via pkgsrc...
mysql-client-5.6.51nb1 MySQL 5, a free SQL database (client)
mysql-connector-c++-8.0.25 Standardized MySQL database driver for C++ development
mysql-server-5.6.51nb1 MySQL 5, a free SQL database (server)
I'd recommend percona-cluster on native, it's kept reasonably up-to-date and supports the wsrep/sst stuff.
