Worker node not coming up with the new kernel after reboot

[ravicorning]

I'm on CentOs 7.8.2003, and using the latest OEK, 20.12.2

Following are the last logs after i run the ./deploy_sh to configure the nodes. Seems the new kernel is taking the host into a bad state...any ideas what might be going wrong ?

Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.10.0-1127.19.1.rt56.1116.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-1127.19.1.rt56.1116.el7.x86_64.img Found linux image: /boot/vmlinuz-3.10.0-1127.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-1127.el7.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-cd2908905321447789efff0871aa4a65 Found initrd image: /boot/initramfs-0-rescue-cd2908905321447789efff0871aa4a65.img done

TASK [machine_setup/grub : get current CMDLINE] ******************************************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/grub/tasks/main.yml:40 ok: [node01] => { "changed": false, "cmd": [ "cat", "/proc/cmdline" ], "delta": "0:00:00.004990", "end": "2021-03-04 13:55:11.746850", "rc": 0, "start": "2021-03-04 13:55:11.741860" }

STDOUT:

BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

TASK [machine_setup/grub : request reboot (will be performed by conditional_reboot role)] ************************************************************************************ task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/grub/tasks/main.yml:46 ok: [node01] => { "ansible_facts": { "perform_reboot": [ "kernel", "grub" ] }, "changed": false }

TASK [machine_setup/configure_tuned : install tuned packages] ****************************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:5 ok: [node01] => { "attempts": 1, "changed": false, "rc": 0, "results": [ "tuned-2.11.0-8.el7.noarch providing tuned-2.11.0-8.el7 is already installed", "tuned-profiles-realtime-2.11.0-8.el7.noarch providing /root/.ansible/tmp/ansible-moduletmp-1614894912.58-33w9XE/tuned-profiles-realtime-2.11.0-8.el7.noarchV2C6y2.rpm is already installed" ] }

TASK [machine_setup/configure_tuned : install tuned packages - offline mode] *************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:13 skipping: [node01] => { "changed": false, "skip_reason": "Conditional result was False" }

TASK [machine_setup/configure_tuned : find isolcpus value if applied as grub parameter] **************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:23 skipping: [node01] => { "changed": false, "skip_reason": "Conditional result was False" }

TASK [machine_setup/configure_tuned : change isolcpus tuned setting] *********************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:26 skipping: [node01] => { "changed": false, "skip_reason": "Conditional result was False" }

TASK [machine_setup/configure_tuned : copy tuned variables into the file] ****************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:31 changed: [node01] => { "changed": true, "checksum": "3de7712038e7523848d66a80cc78bbed32d726a3", "dest": "/etc/tuned/realtime-variables.conf", "gid": 0, "group": "root", "md5sum": "fd84ed0a56591d879fddd2265ca1954b", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:tuned_etc_t:s0", "size": 41, "src": "/root/.ansible/tmp/ansible-tmp-1614894881.43-21337-212133244989112/source", "state": "file", "uid": 0 }

TASK [machine_setup/configure_tuned : apply tuned profile] ********************************************************************************************************************* task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:40 changed: [node01] => { "changed": true, "cmd": [ "tuned-adm", "profile", "realtime" ], "delta": "0:00:02.487842", "end": "2021-03-04 13:55:17.010233", "rc": 0, "start": "2021-03-04 13:55:14.522391" }

TASK [machine_setup/configure_tuned : get tuned CMDLINE] *********************************************************************************************************************** task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:45 ok: [node01] => { "changed": false, "cmd": [ "awk", "-F"", "/TUNED_BOOT_CMDLINE/ {print $2}", "/etc/tuned/bootcmdline" ], "delta": "0:00:00.002240", "end": "2021-03-04 13:55:17.285015", "rc": 0, "start": "2021-03-04 13:55:17.282775" }

STDOUT:

skew_tick=1 isolcpus=2-3 intel_pstate=disable nosoftlockup

TASK [machine_setup/configure_tuned : get current CMDLINE] ********************************************************************************************************************* task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:50 ok: [node01] => { "changed": false, "cmd": [ "cat", "/proc/cmdline" ], "delta": "0:00:00.001627", "end": "2021-03-04 13:55:17.544039", "rc": 0, "start": "2021-03-04 13:55:17.542412" }

STDOUT:

BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8

TASK [machine_setup/configure_tuned : request reboot (will be performed by conditional_reboot role)] ************************************************************************* task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/configure_tuned/tasks/main.yml:56 ok: [node01] => { "ansible_facts": { "perform_reboot": [ "kernel", "grub", "tuned" ] }, "changed": false }

TASK [machine_setup/conditional_reboot : display reboot requesters] ************************************************************************************************************ task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/conditional_reboot/tasks/main.yml:8 ok: [node01] => {}

MSG:

Performing reboot requested by: [u'kernel', u'grub', u'tuned']

TASK [machine_setup/conditional_reboot : reboot the machine] ******************************************************************************************************************* task path: /home/ravi/OpenNess-20.12.02/openness-experience-kits/roles/machine_setup/conditional_reboot/tasks/main.yml:11

[aniket-intel]

Hi Ravi,

Can you please try removing mce=off from the file oek/flavors/flexran/edgenode_group.yml in line 35, and try the deploy script again after cleaning up the system? Let me know if that works.