initscripts icon indicating copy to clipboard operation
initscripts copied to clipboard
verified publisher icon fedora-sysv

Allow to slave devices execute ifup-post.

Open k0ste opened this issue 8 years ago • 10 comments

Hello. I find out that slave devices in bond LACP LAG can't be tuned via ifup-local. As I understand the exit after SLAVE key is for safety (ignore another options). So for save this behaviour I just added exec for ifup-post.

Work of this patch tested on various baremetal servers. For example:

#!/bin/bash
case "$1" in
ten0)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten1)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten2)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
ten3)
echo -e "Set ring params for ${1}:\\n"
ethtool -G $1 rx 4096 tx 4096
sleep 0.5s
echo -e "Set features for ${1}:\\n"
ethtool -K $1 gso off tso off gro off rx off tx off sg off txvlan off rxvlan off
sleep 0.5s
echo -e "Interrupt moderation for ${1}:\\n"
ethtool -C $1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
sleep 0.5s
tc qdisc add dev $1 root fq_codel
ip link set dev $1 txqueuelen 30000
;;
vlan2)
sleep 1s
/usr/bin/irq-tune -j /etc/irq-tune/irq_static.json
/usr/bin/irq-tune -a -j /etc/irq-tune/irq_auto.json
;;
esac
exit 0

journal log:

-- Reboot --
Jun 18 16:17:40 ceph-osd0 systemd[1]: Starting LSB: Bring up/down networking...
Jun 18 16:17:40 ceph-osd0 network[623]: Bringing up loopback interface:  [  OK  ]
Jun 18 16:17:41 ceph-osd0 network[623]: Bringing up interface bond0:  Set ring params for ten0:
Jun 18 16:17:41 ceph-osd0 network[623]: Set features for ten0:
Jun 18 16:17:42 ceph-osd0 network[623]: Interrupt moderation for ten0:
Jun 18 16:17:42 ceph-osd0 network[623]: Set ring params for ten1:
Jun 18 16:17:43 ceph-osd0 network[623]: Set features for ten1:
Jun 18 16:17:43 ceph-osd0 network[623]: Interrupt moderation for ten1:
Jun 18 16:17:44 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:44 ceph-osd0 network[623]: Bringing up interface bond1:  Set ring params for ten2:
Jun 18 16:17:45 ceph-osd0 network[623]: Set features for ten2:
Jun 18 16:17:45 ceph-osd0 network[623]: Interrupt moderation for ten2:
Jun 18 16:17:46 ceph-osd0 network[623]: Set ring params for ten3:
Jun 18 16:17:46 ceph-osd0 network[623]: Set features for ten3:
Jun 18 16:17:47 ceph-osd0 network[623]: Interrupt moderation for ten3:
Jun 18 16:17:47 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:53 ceph-osd0 network[623]: Bringing up interface vlan2:  Set IRQ [megasas]: 27 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/27/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 28 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/28/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 29 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/29/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 30 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/30/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 31 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/31/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [megasas]: 33 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/33/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: ****** ERROR ****** The IRQ 34, for megasas does not exists
Jun 18 16:17:53 ceph-osd0 network[623]: ****** ERROR ****** The IRQ 35, for megasas does not exists
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-0]: 41 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/41/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-1]: 42 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/42/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-2]: 43 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/43/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-3]: 44 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/44/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-4]: 45 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/45/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-5]: 46 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/46/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-6]: 47 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/47/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten0-TxRx-7]: 48 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/48/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-0]: 60 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/60/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-1]: 61 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/61/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-2]: 62 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/62/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-3]: 63 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/63/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-4]: 64 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/64/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-5]: 65 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/65/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-6]: 66 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/66/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten1-TxRx-7]: 67 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/67/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-0]: 78 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/78/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-1]: 79 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/79/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-2]: 80 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/80/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-3]: 81 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/81/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-4]: 82 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/82/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-5]: 83 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/83/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-6]: 84 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/84/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten2-TxRx-7]: 85 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/85/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-0]: 96 to cores: 7
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000080 procfile: /proc/irq/96/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-1]: 97 to cores: 6
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000040 procfile: /proc/irq/97/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-2]: 98 to cores: 5
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000020 procfile: /proc/irq/98/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-3]: 99 to cores: 4
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000010 procfile: /proc/irq/99/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-4]: 100 to cores: 3
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000008 procfile: /proc/irq/100/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-5]: 101 to cores: 2
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000004 procfile: /proc/irq/101/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-6]: 102 to cores: 1
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000002 procfile: /proc/irq/102/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: Set IRQ [i40e-ten3-TxRx-7]: 103 to cores: 0
Jun 18 16:17:53 ceph-osd0 network[623]: mask: 00000000,00000001 procfile: /proc/irq/103/smp_affinity
Jun 18 16:17:53 ceph-osd0 network[623]: [  OK  ]
Jun 18 16:17:57 ceph-osd0 network[623]: Bringing up interface vlan3:  [  OK  ]
Jun 18 16:18:01 ceph-osd0 network[623]: Bringing up interface vlan4:  [  OK  ]
Jun 18 16:18:01 ceph-osd0 systemd[1]: Started LSB: Bring up/down networking.

k0ste avatar Jul 20 '17 05:07 k0ste

Hello @k0ste, we were looking into you problem description, but we do not exactly understand what's your issue...

I find out that slave devices in bond LACP LAG can't be tuned via ifup-local.

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you.

deekej avatar Jul 24 '17 16:07 deekej

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you.

I provided example ifup-local script and stdout of this script work ("proof of concept").

k0ste avatar Jul 25 '17 00:07 k0ste

What exactly can't be "tuned" in your case? Please, provide us with more information. Thank you. I provided example ifup-local script and stdout of this script work ("proof of concept").

Hmm, can't you set most of the bits from your example via ETHTOOL_OPTS?

The reason why we ask is that we would like to find a different way how to implement the behavior you want. Calling ifup-post here is not an option, it does much more than just calling ifup-local and this could definitely break someone else's setup. I would also like to avoid directly calling ifup-local here, because again I can imagine a use-case it could break.

lnykryn avatar Jul 25 '17 08:07 lnykryn

Hmm, can't you set most of the bits from your example via ETHTOOL_OPTS?

Nope, because sometimes (various of device) we need timeout between ethtool calls, when driver reloaded device.

If we try do this jobs on MASTER interface (i.e. bond0) with switches that unsupported LACP rate fast (like Cisco Catalyst 29**) - possible break of LAG, due timeouts without any slave.

With i40e is pretty fast:

[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: changing MTU from 1500 to 9000
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: already using mac address 3c:fd:fe:9f:ea:0c
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: Changing Tx descriptor count from 512 to 4096.
[Sat Jun 17 18:40:51 2017] i40e 0000:02:00.0 ten0: Changing Rx descriptor count from 512 to 4096
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: changing MTU from 1500 to 9000
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: set new mac address 3c:fd:fe:9f:ea:0c
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: NIC Link is Up 10 Gbps Full Duplex, Flow Control: None
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: Changing Tx descriptor count from 512 to 4096.
[Sat Jun 17 18:40:53 2017] i40e 0000:02:00.1 ten1: Changing Rx descriptor count from 512 to 4096

But with tg3 we need more time:

[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.0 eth0: EEE is disabled          
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:15 2017] tg3 0000:03:00.1 eth1: EEE is disabled          
[Sat Jul  8 18:35:17 2017] tg3 0000:03:00.0 eth0: speed changed to 0 for port eth0                                                                    
[Sat Jul  8 18:35:19 2017] tg3 0000:03:00.1 eth1: speed changed to 0 for port eth1                                                                    
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:20 2017] tg3 0000:03:00.0 eth0: EEE is disabled          
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: Link is up at 1000 Mbps, full duplex                                                                
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: Flow control is off for TX and off for RX                                                           
[Sat Jul  8 18:35:22 2017] tg3 0000:03:00.1 eth1: EEE is disabled

k0ste avatar Jul 25 '17 08:07 k0ste

We already have ETHTOOL_DELAY which waits before calling ethtool, maybe we could also add it to the for loop. Something like:

diff --git a/sysconfig/network-scripts/network-functions b/sysconfig/network-scripts/network-functions
index d72901e..fbbb326 100644
--- a/sysconfig/network-scripts/network-functions
+++ b/sysconfig/network-scripts/network-functions
@@ -209,6 +209,7 @@ ethtool_set()
             /sbin/ethtool -s ${REALDEVICE} $opts
         fi
         IFS=';';
+        sleep ${ETHTOOL_DELAY_SEC}
     done
     IFS=$oldifs;
 }

lnykryn avatar Jul 25 '17 10:07 lnykryn

But looking to your script, you do only one ethtool call per device, so the current ETHTOOL_DELAY should be working for you.

lnykryn avatar Jul 25 '17 10:07 lnykryn

But looking to your script, you do only one ethtool call per device, so the current ETHTOOL_DELAY should be working for you.

Why you ignore echo, iproute & traffic control commands? If you do not configure your physical network interfaces or run hooks like apply htb and sfq queues, this does not mean that someone does not use such hacks, I'm just tired of apply this patch on my CentOS nodes.

k0ste avatar Jul 25 '17 12:07 k0ste

I created issue #153, because don't see any movements about this PR.

k0ste avatar Nov 15 '17 09:11 k0ste

Well, I currently do not have any solution which would satisfy your needs and wouldn't break any existing user setups. Do you? :)

deekej avatar Nov 20 '17 12:11 deekej

Well, I currently do not have any solution which would satisfy your needs and wouldn't break any existing user setups.

Please, explain for me: how it break existing user setups? May be I don't understand something and will search another solution.

k0ste avatar Nov 20 '17 13:11 k0ste

network-scripts are retired /F

k0ste avatar Feb 20 '25 09:02 k0ste