Starting tuned service breaks ipv6

Open nerijus opened this issue 2 years ago • 1 comments

tuned 2.20.0, CentOS 9. On one server restarting service is OK:

systemctl restart tuned

Mar 26 21:01:26 server01 systemd[1]: Stopping Dynamic System Tuning Daemon...
Mar 26 21:01:26 server01 systemd[1]: tuned.service: Deactivated successfully.
Mar 26 21:01:26 server01 systemd[1]: Stopped Dynamic System Tuning Daemon.
Mar 26 21:01:26 server01 systemd[1]: Starting Dynamic System Tuning Daemon...
Mar 26 21:01:26 server01 systemd[1]: Started Dynamic System Tuning Daemon.

On another:

systemctl restart tuned

Mar 26 21:11:12 server02 systemd[1]: Stopping Dynamic System Tuning Daemon...
Mar 26 21:11:12 server02 systemd[1]: tuned.service: Deactivated successfully.
Mar 26 21:11:12 server02 systemd[1]: Stopped Dynamic System Tuning Daemon.
Mar 26 21:11:12 server02 systemd[1]: Starting Dynamic System Tuning Daemon...
Mar 26 21:11:12 server02 NetworkManager[896]: <warn>  [1679865072.7545] platform-linux: do-add-ip6-address[2: fe80::caec:19f1:caa6:dda1]: failure 13 (Permission denied)
Mar 26 21:11:12 server02 systemd[1]: Started Dynamic System Tuning Daemon.
Mar 26 21:11:14 server02 NetworkManager[896]: <warn>  [1679865074.7568] platform-linux: do-add-ip6-address[2: fe80::d5d0:a6f3:8779:1ee6]: failure 13 (Permission denied)
Mar 26 21:11:16 server02 NetworkManager[896]: <warn>  [1679865076.7590] platform-linux: do-add-ip6-address[2: fe80::bb3b:7344:222d:7214]: failure 13 (Permission denied)
Mar 26 21:11:18 server02 NetworkManager[896]: <warn>  [1679865078.7612] platform-linux: do-add-ip6-address[2: fe80::3639:34e1:87e5:983f]: failure 13 (Permission denied)
Mar 26 21:11:20 server02 NetworkManager[896]: <warn>  [1679865080.7634] platform-linux: do-add-ip6-address[2: fe80::a27b:239b:1b20:491e]: failure 13 (Permission denied)
Mar 26 21:11:22 server02 NetworkManager[896]: <warn>  [1679865082.7656] ipv6ll[a552faf38ec5f878,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off)
Mar 26 21:11:22 server02 NetworkManager[896]: <warn>  [1679865082.7657] platform: (enp41s0) route-sync: failure to add IPv6 route: type unicast fe80::1/128 dev 2 metric 100 mss 0 rt-src user: Permission denied

After this on 2nd server ipv6 is lost:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp41s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:a1:59:a2:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 65.108.xx.xx/24 brd 65.108.xx.255 scope global noprefixroute enp41s0
       valid_lft forever preferred_lft forever

It reappears after systemctl restart NetworkManager:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp41s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:a1:59:a2:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 65.108.xx.xx/24 brd 65.108.xx.255 scope global noprefixroute enp41s0
       valid_lft forever preferred_lft forever
    inet6 2a01:xxx:1a:xxxx::2/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::a16d:5110:fc52:2a8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Mar 27 '23 08:03 nerijus

/var/log/tuned/tuned.log from server01:

2023-03-27 10:10:23,779 INFO     tuned.daemon.controller: terminating controller
2023-03-27 10:10:23,779 INFO     tuned.daemon.daemon: stopping tuning
2023-03-27 10:10:23,787 INFO     tuned.daemon.daemon: terminating TuneD, rolling back all changes
2023-03-27 10:10:23,788 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu17'
...
2023-03-27 10:10:23,790 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'normal' on cpu 'cpu14'
2023-03-27 10:10:23,791 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu17'
...
2023-03-27 10:10:23,793 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu14'
2023-03-27 10:10:23,850 INFO     tuned.daemon.application: TuneD: 2.20.0, kernel: 5.14.0-289.el9.x86_64
2023-03-27 10:10:23,851 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2023-03-27 10:10:23,852 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2023-03-27 10:10:23,852 INFO     tuned.profiles.loader: loading profile: throughput-performance
2023-03-27 10:10:23,854 INFO     tuned.daemon.controller: starting controller
2023-03-27 10:10:23,854 INFO     tuned.daemon.daemon: starting tuning
2023-03-27 10:10:23,872 INFO     tuned.plugins.base: instance cpu: assigning devices cpu13, cpu4, cpu19, cpu2, cpu7, cpu1, cpu10, cpu17, cpu5, cpu18, cpu8, cpu16, cpu0, cpu15, cpu11, cpu3, cpu14, cpu12, cpu9, cpu6
2023-03-27 10:10:23,885 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform
2023-03-27 10:10:23,887 INFO     tuned.plugins.plugin_cpu: intel_pstate detected
2023-03-27 10:10:23,889 INFO     tuned.plugins.plugin_disk: Device 'nvme0n1' not supported by hdparm
2023-03-27 10:10:23,890 INFO     tuned.plugins.plugin_disk: Device 'nvme1n1' not supported by hdparm
2023-03-27 10:10:23,891 INFO     tuned.plugins.plugin_disk: Device 'md0' not supported by hdparm
2023-03-27 10:10:23,892 INFO     tuned.plugins.plugin_disk: Device 'md127' not supported by hdparm
2023-03-27 10:10:23,893 INFO     tuned.plugins.plugin_disk: Device 'md2' not supported by hdparm
2023-03-27 10:10:23,894 INFO     tuned.plugins.base: instance disk: assigning devices nvme0n1, nvme1n1, md127, md2, md0
2023-03-27 10:10:23,911 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu13'
...
2023-03-27 10:10:23,914 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu8'
2023-03-27 10:10:23,914 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu13'
...
2023-03-27 10:10:23,916 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu8'
2023-03-27 10:10:23,917 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2023-03-27 10:10:23,917 INFO     tuned.daemon.daemon: static tuning from profile 'throughput-performance' applied

server02:

2023-03-27 10:10:48,647 INFO     tuned.daemon.controller: terminating controller
2023-03-27 10:10:48,647 INFO     tuned.daemon.daemon: stopping tuning
2023-03-27 10:10:48,657 INFO     tuned.daemon.daemon: terminating TuneD, rolling back all changes
2023-03-27 10:10:48,657 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu11'
...
2023-03-27 10:10:48,659 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu8'
2023-03-27 10:10:48,738 INFO     tuned.daemon.application: TuneD: 2.20.0, kernel: 5.14.0-289.el9.x86_64
2023-03-27 10:10:48,738 INFO     tuned.daemon.application: dynamic tuning is globally disabled
2023-03-27 10:10:48,740 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
2023-03-27 10:10:48,741 INFO     tuned.profiles.loader: loading profile: throughput-performance
2023-03-27 10:10:48,742 INFO     tuned.daemon.controller: starting controller
2023-03-27 10:10:48,742 INFO     tuned.daemon.daemon: starting tuning
2023-03-27 10:10:48,760 INFO     tuned.plugins.base: instance cpu: assigning devices cpu4, cpu0, cpu2, cpu6, cpu10, cpu3, cpu11, cpu7, cpu9, cpu5, cpu1, cpu8
2023-03-27 10:10:48,773 INFO     tuned.plugins.plugin_cpu: We are running on an x86 AuthenticAMD platform
2023-03-27 10:10:48,776 INFO     tuned.plugins.plugin_disk: Device 'nvme0n1' not supported by hdparm
2023-03-27 10:10:48,778 INFO     tuned.plugins.plugin_disk: Device 'nvme1n1' not supported by hdparm
2023-03-27 10:10:48,780 INFO     tuned.plugins.plugin_disk: Device 'md126' not supported by hdparm
2023-03-27 10:10:48,781 INFO     tuned.plugins.plugin_disk: Device 'md127' not supported by hdparm
2023-03-27 10:10:48,781 INFO     tuned.plugins.base: instance disk: assigning devices nvme0n1, nvme1n1, md126, md127
2023-03-27 10:10:48,782 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu4'
...
2023-03-27 10:10:48,784 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu8'
2023-03-27 10:10:48,786 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
2023-03-27 10:10:48,788 INFO     tuned.daemon.daemon: static tuning from profile 'throughput-performance' applied

Mar 27 '23 10:03 nerijus