Flatcar icon indicating copy to clipboard operation
Flatcar copied to clipboard

Published 20 hours ago • verified publisher icon flatcar

multipathd fails to start after 3139.2.3 upgrade to 3227.2.0

Open defo89 opened this issue 2 years ago • 6 comments

Description

When 3139.2.3 is upgraded to 3227.2.0, multipathd service fails to start. Flatcar deployed with OVA template to vSphere 7.0.3.

Impact

Server is available only after ~50 minutes after reaching the timeout to start multipathd.

Environment and steps to reproduce

  1. Set-up: server running 3139.2.3 with multipathd enabled.
cat /etc/multipath.conf
defaults {
  user_friendly_names yes
  find_multipaths yes
}

ignition-v2 file to reproduce:

{
    "ignition": {
      "config": {},
      "timeouts": {},
      "version": "2.1.0"
    },
    "storage": {
      "files": [
        {
          "filesystem": "root",
          "group": {},
          "overwrite": false,
          "path": "/etc/multipath.conf",
          "user": {},
          "contents": {
            "source": "data:text/plain;charset=utf-8;base64,ZGVmYXVsdHMgewogIHVzZXJfZnJpZW5kbHlfbmFtZXMgeWVzCiAgZmluZF9tdWx0aXBhdGhzIHllcwp9Cg==",
            "verification": {}
          }
        }
      ]
    },
    "systemd": {
      "units": [
        {
          "enabled": true,
          "name": "multipathd.service"
        }
      ]
    }
  }
  1. Task: Update to latest stable 3227.2.0 and reboot
update_engine_client -update
  1. Error:

After ~50 minutes server boots up with failed service:

Flatcar Container Linux by Kinvolk stable 3227.2.0 for VMware
Failed Units: 2
  multipathd.service
  multipathd.socket

flatcar-multipath ~ # systemctl status multipathd
× multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Mon 2022-07-25 13:16:00 UTC; 3min 3s ago
TriggeredBy: × multipathd.socket
    Process: 920 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, status=0/SUCCESS)
    Process: 921 ExecStart=/sbin/multipathd -d -s (code=dumped, signal=ABRT)
   Main PID: 921 (code=dumped, signal=ABRT)
        CPU: 4ms

Journal shows attempts to start the service (during those server is not accessible):

Jul 25 12:27:53 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:27:53 flatcar-multipath multipathd[786]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:29:23 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 786 (multipathd) with signal SIGKILL.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 787 (multipathd) with signal SIGKILL.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:30:53 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.

Additional information

The issue occurs only in the upgrade case. Issue does not occur if server is deployed directly with 3227.2.0.

Looking at differences before/after the upgrade, here is what I was able to gather.

3139.2.3:

multipath-tools v0.8.5 (11/09, 2020)

# find / -name libgcc_s.so.1
/usr/lib64/libgcc_s.so.1

# ldconfig -p | grep libgcc
	libgcc_s.so.1 (libc6,x86-64) => /lib64/libgcc_s.so.1

3227.2.0:

multipath-tools v0.8.7 (09/08, 2021)

# find / -name libgcc_s.so.1
/usr/lib/libgcc_s.so.1

# ldconfig -p | grep libgcc
	libgcc_s.so.1 (libc6,x86-64) => /usr/lib/libgcc_s.so.1

Service will run successfully after manual start (systemctl start multipathd) or server reboot. Main concern is prolonged server downtime during the upgrade and possibility to avoid such occurrences in the next upgrade (for this or any other service).

defo89 avatar Jul 25 '22 13:07 defo89

full log

flatcar-multipath ~ # journalctl --no-pager --boot -u multipathd
Jul 25 12:27:53 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:27:53 flatcar-multipath multipathd[786]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:29:23 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 786 (multipathd) with signal SIGKILL.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 787 (multipathd) with signal SIGKILL.
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:30:53 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:30:53 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:30:53 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:30:53 flatcar-multipath multipathd[790]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:32:23 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:33:53 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:33:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 790 (multipathd) with signal SIGKILL.
Jul 25 12:33:53 flatcar-multipath systemd[1]: multipathd.service: Killing process 791 (multipathd) with signal SIGKILL.
Jul 25 12:33:53 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:33:53 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:33:53 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:33:53 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:33:53 flatcar-multipath multipathd[799]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:35:24 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:36:54 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:36:54 flatcar-multipath systemd[1]: multipathd.service: Killing process 799 (multipathd) with signal SIGKILL.
Jul 25 12:36:54 flatcar-multipath systemd[1]: multipathd.service: Killing process 800 (multipathd) with signal SIGKILL.
Jul 25 12:36:54 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:36:54 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:36:54 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:36:54 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:36:54 flatcar-multipath multipathd[804]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:38:24 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:39:54 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:39:54 flatcar-multipath systemd[1]: multipathd.service: Killing process 804 (multipathd) with signal SIGKILL.
Jul 25 12:39:54 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:39:54 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:39:54 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:39:54 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:39:54 flatcar-multipath multipathd[813]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:41:25 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:42:55 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:42:55 flatcar-multipath systemd[1]: multipathd.service: Killing process 813 (multipathd) with signal SIGKILL.
Jul 25 12:42:55 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:42:55 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:42:55 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:42:55 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:42:55 flatcar-multipath multipathd[818]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:44:25 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:45:55 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:45:55 flatcar-multipath systemd[1]: multipathd.service: Killing process 818 (multipathd) with signal SIGKILL.
Jul 25 12:45:55 flatcar-multipath systemd[1]: multipathd.service: Killing process 819 (multipathd) with signal SIGKILL.
Jul 25 12:45:55 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:45:55 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:45:55 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:45:55 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:45:55 flatcar-multipath multipathd[826]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:47:26 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:48:56 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:48:56 flatcar-multipath systemd[1]: multipathd.service: Killing process 826 (multipathd) with signal SIGKILL.
Jul 25 12:48:56 flatcar-multipath systemd[1]: multipathd.service: Killing process 827 (multipathd) with signal SIGKILL.
Jul 25 12:48:56 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:48:56 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:48:56 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:48:56 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:48:56 flatcar-multipath multipathd[834]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:50:26 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:51:56 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:51:56 flatcar-multipath systemd[1]: multipathd.service: Killing process 834 (multipathd) with signal SIGKILL.
Jul 25 12:51:56 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:51:56 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:51:56 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:51:56 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:51:56 flatcar-multipath multipathd[840]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:53:27 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:54:57 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:54:57 flatcar-multipath systemd[1]: multipathd.service: Killing process 840 (multipathd) with signal SIGKILL.
Jul 25 12:54:57 flatcar-multipath systemd[1]: multipathd.service: Killing process 841 (multipathd) with signal SIGKILL.
Jul 25 12:54:57 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:54:57 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:54:57 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:54:57 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:54:57 flatcar-multipath multipathd[848]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:56:27 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 12:57:57 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 12:57:57 flatcar-multipath systemd[1]: multipathd.service: Killing process 848 (multipathd) with signal SIGKILL.
Jul 25 12:57:57 flatcar-multipath systemd[1]: multipathd.service: Killing process 849 (multipathd) with signal SIGKILL.
Jul 25 12:57:57 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 12:57:57 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 12:57:57 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 12:57:57 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 12:57:57 flatcar-multipath multipathd[854]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 12:59:28 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:00:58 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:00:58 flatcar-multipath systemd[1]: multipathd.service: Killing process 854 (multipathd) with signal SIGKILL.
Jul 25 13:00:58 flatcar-multipath systemd[1]: multipathd.service: Killing process 856 (n/a) with signal SIGKILL.
Jul 25 13:00:58 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:00:58 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:00:58 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:00:58 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:00:58 flatcar-multipath multipathd[863]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:02:28 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:03:58 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:03:58 flatcar-multipath systemd[1]: multipathd.service: Killing process 863 (multipathd) with signal SIGKILL.
Jul 25 13:03:58 flatcar-multipath systemd[1]: multipathd.service: Killing process 864 (multipathd) with signal SIGKILL.
Jul 25 13:03:58 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:03:58 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:03:58 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:03:58 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:03:58 flatcar-multipath multipathd[873]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:05:29 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:06:59 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:06:59 flatcar-multipath systemd[1]: multipathd.service: Killing process 873 (multipathd) with signal SIGKILL.
Jul 25 13:06:59 flatcar-multipath systemd[1]: multipathd.service: Killing process 874 (n/a) with signal SIGKILL.
Jul 25 13:06:59 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:06:59 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:06:59 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:06:59 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:06:59 flatcar-multipath multipathd[877]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:08:29 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:09:59 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:09:59 flatcar-multipath systemd[1]: multipathd.service: Killing process 877 (multipathd) with signal SIGKILL.
Jul 25 13:09:59 flatcar-multipath systemd[1]: multipathd.service: Killing process 879 (multipathd) with signal SIGKILL.
Jul 25 13:09:59 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:09:59 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:09:59 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:09:59 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:09:59 flatcar-multipath multipathd[887]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:11:30 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:13:00 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:13:00 flatcar-multipath systemd[1]: multipathd.service: Killing process 887 (multipathd) with signal SIGKILL.
Jul 25 13:13:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:13:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:13:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:13:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:13:00 flatcar-multipath multipathd[892]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:14:30 flatcar-multipath systemd[1]: multipathd.service: start operation timed out. Terminating.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: State 'stop-sigterm' timed out. Killing.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Killing process 892 (multipathd) with signal SIGKILL.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Killing process 893 (multipathd) with signal SIGKILL.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=killed, status=9/KILL
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'timeout'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:16:00 flatcar-multipath multipathd[900]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:16:00 flatcar-multipath multipathd[907]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:16:00 flatcar-multipath multipathd[912]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:16:00 flatcar-multipath multipathd[917]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jul 25 13:16:00 flatcar-multipath multipathd[921]: libgcc_s.so.1 must be installed for pthread_cancel to work
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Main process exited, code=dumped, status=6/ABRT
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Start request repeated too quickly.
Jul 25 13:16:00 flatcar-multipath systemd[1]: multipathd.service: Failed with result 'core-dump'.
Jul 25 13:16:00 flatcar-multipath systemd[1]: Failed to start Device-Mapper Multipath Device Controller.

defo89 avatar Jul 25 '22 13:07 defo89

journalctl-b0.log

defo89 avatar Jul 25 '22 13:07 defo89

ldconfig cache is out of date, and needs an update after the "/lib"/"/lib64" split. I'll check if this is the only issue and then see how to get this done during/after the update.

jepio avatar Jul 25 '22 14:07 jepio

This is a blocker for our on-prem cluster as well. Is there a workaround that would allow us to upgrade to the latest flatcar image, while we wait for fixes to be merged into a new release?

ffilippopoulos avatar Jul 28 '22 10:07 ffilippopoulos

Create this file before rebooting /etc/systemd/system/multipathd.service.d/pre.conf:

[Service]
ExecStartPre=ldconfig

This should work as a workaround, normally ldconfig will run during boot after an update but it runs after multipathd starts, and there would be an ordering cycle if one were to add a dependency between multipathd.service and ldconfig.service (the ordering is already: multipathd.service -> local-fs-pre.target -> local-fs.target -> ldconfig.service).

We plan to have a bugfix release out around next week.

jepio avatar Jul 28 '22 11:07 jepio

tyvm @jepio !

ffilippopoulos avatar Jul 28 '22 11:07 ffilippopoulos

Fix has been released, I guess we can close this issue. Thanks @defo89 for the initial report!

tormath1 avatar Nov 10 '22 09:11 tormath1