pacemaker
pacemaker copied to clipboard
ClusterLabs
Doc: pcmk_action_timeout update
The default pcmk_<action>_timeout values listed in the doc are
incorrect in my experience. The monitor timeout defaults to 20s, and all
the rest default to 120s.
Also, lib/pengine/common.c states that stonith-timeout is now
unused. If so, then pcmk_<action>_timeout shouldn't be described as
an alternative or override to stonith-timeout.
Correcting the fencing timeout documentation has been on my to-do for a while ...
Going through the code again, it looks like the documentation is even more wrong.
The stonith-timeout cluster property is in fact used:
- The scheduler unpacks it into data_set->stonith_timeout via unpack_config(), using the definition in pe_opts[] (default 60s)
- The scheduler adds the value to the transition graph via stage8()
- The controller unpacks it from the transition graph via unpack_graph()
- The controller uses it as the initial timeout for all cluster-initiated fence actions via te_fence_node()
Separately, both Pacemaker Explained and stonith_api_validate() treat stonith-timeout as a Pacemaker-supplied fencing resource parameter. Pacemaker itself does not use or set this at all, but I believe some Linux-HA-style fencing agents (which use fence_legacy) use it. Perhaps the documentation is correct for those agents.
As far as I can tell, there is technically no default for most stonith operations; the client must specify a value when contacting the fencer.
- stonith_admin uses a default of 120s (in its options struct)
- the controller uses stonith-timeout as described above
- the fencer uses a default of 20s (DEFAULT_QUERY_TIMEOUT) for internally scheduled list and status commands (for dynamic target detection), if the client didn't specify a timeout or specified one that was too low
- other clients (e.g. dlm) would have their own defaults
There is a wrinkle: a stonith action such as reboot might actually consist of multiple operations due to a topology or retries. Thus the fencer calculates a total timeout that is appropriate to the entire action, and informs the client of the new value via a callback (which is how the controller knows not to timeout the action if it's longer than the originally passed timeout).
Recurring monitors are different. They have two configurable timeouts, the pcmk_monitor_timeout and the CIB recurring op timeout. I'm not sure how that works; possibly the lower one wins, or possibly the CIB timeout is always used, or possibly the pcmk_monitor_timeout is used for any one attempt and the CIB timeout covers the entire action (including retries). It's likely that pcmk_monitor_timeout defaults the same as the other fencing op defaults, but the CIB timeout follows the usual operation defaults.
How to document all that without driving the user mad is an open question ;)
The stonith-timeout cluster property is in fact used:
It looked that way to me too just from the code. I didn't get around to testing the property, as this was a secondary concern at the time. That's why I was careful to say "If so" re: Pacemaker not using stonith-timeout :) Good to see it confirmed, even though that does make things stickier.
Pacemaker itself does not use or set this at all, but I believe some Linux-HA-style fencing agents (which use fence_legacy) use it.
Maybe. I've never personally encountered a fence agent that uses stonith-timeout, and I can't find any reference to it in the current fence-agents upstream besides an incorrect example command: https://github.com/ClusterLabs/fence-agents/blob/master/agents/rcd_serial/fence_rcd_serial.py#L12
the fencer uses a default of 20s (DEFAULT_QUERY_TIMEOUT) for internally scheduled list and status commands (for dynamic target detection), if the client didn't specify a timeout or specified one that was too low
I had tested list and status with stonith_admin previously. Now, with internally scheduled list and status commands, I'm seeing 60s.
# with pcmk_host_check=status
Resource: vmfence (class=stonith type=fence_vmware_soap)
Attributes: pcmk_host_check=status pcmk_monitor_retries=1 pcmk_monitor_timeout=30s pcmk_reboot_retries=1 ssl_insecure=1
Jul 16 12:08:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (schedule_stonith_command) debug: Scheduling 'status' action targeting fastvm-rhel-8-0-24 on vmfence for pacemaker-fenced (timeout=60s)
Jul 16 12:08:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (stonith_command) debug: Processed st_query from fastvm-rhel-8-0-23: OK (0)
Jul 16 12:08:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (make_args) debug: Performing 'status' action targeting 'fastvm-rhel-8-0-24' as 'port=fastvm-rhel-8-0-24'
Jul 16 12:08:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (stonith_action_create) debug: Preparing 'status' action for fastvm-rhel-8-0-24 using agent fence_vmware_soap
...
Jul 16 12:09:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (child_timeout_callback) warning: fence_vmware_soap_status_1 process (PID 1211115) timed out
Jul 16 12:09:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (operation_finished) warning: fence_vmware_soap_status_1[1211115] timed out after 60000ms
Jul 16 12:09:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (stonith_action_async_done) debug: Child process 1211115 performing action 'status' exited with rc 198
Jul 16 12:09:29 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (remote_op_done) error: Operation 'reboot' targeting fastvm-rhel-8-0-24 on <no-one> for [email protected]: No such device
Jul 16 12:09:29 fastvm-rhel-8-0-23 pacemaker-controld [1211063] (tengine_stonith_notify) notice: Peer fastvm-rhel-8-0-24 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.1211063: No such device | initiator=fastvm-rhel-8-0-23 ref=9485c7fd-1e5a-4622-9204-a95dd5a4ffa8
# with pcmk_host_check=dynamic-list
Resource: vmfence (class=stonith type=fence_vmware_soap)
Attributes: pcmk_host_check=dynamic-list pcmk_monitor_retries=1 pcmk_monitor_timeout=30s pcmk_reboot_retries=1 ssl_insecure=1
Jul 16 12:15:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (schedule_stonith_command) debug: Scheduling 'list' action on vmfence for pacemaker-fenced (timeout=60s)
Jul 16 12:15:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (stonith_command) debug: Processed st_query from fastvm-rhel-8-0-23: OK (0)
Jul 16 12:15:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (stonith_action_create) debug: Preparing 'list' action for no target using agent fence_vmware_soap
...
Jul 16 12:16:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (child_timeout_callback) warning: fence_vmware_soap_list_1 process (PID 1211192) timed out
Jul 16 12:16:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (operation_finished) warning: fence_vmware_soap_list_1[1211192] timed out after 60000ms
Jul 16 12:16:26 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (remote_op_done) error: Operation 'reboot' targeting fastvm-rhel-8-0-24 on <no-one> for [email protected]: No such device
Jul 16 12:16:26 fastvm-rhel-8-0-23 pacemaker-controld [1211063] (tengine_stonith_notify) notice: Peer fastvm-rhel-8-0-24 was not terminated (reboot) by <anyone> on behalf of pacemaker-controld.1211063: No such device | initiator=fastvm-rhel-8-0-23 ref=8491e45a-085a-45a2-8973-145d48549722
other clients (e.g. dlm) would have their own defaults
DLM, gross ;)
Recurring monitors are different. They have two configurable timeouts, the pcmk_monitor_timeout and the CIB recurring op timeout. I'm not sure how that works; possibly the lower one wins, or possibly the CIB timeout is always used, or possibly the pcmk_monitor_timeout is used for any one attempt and the CIB timeout covers the entire action (including retries). It's likely that pcmk_monitor_timeout defaults the same as the other fencing op defaults, but the CIB timeout follows the usual operation defaults.
The lower one doesn't win. pcmk_monitor_timeout takes over if it's set. Otherwise, the CRM_meta_timeout (which defaults to 20s) is used.
# with pcmk_monitor_timeout=30s
Resource: vmfence (class=stonith type=fence_vmware_soap)
Attributes: <opts> pcmk_monitor_retries=1 pcmk_monitor_timeout=30s pcmk_reboot_retries=1 ssl_insecure=1
Jul 16 12:30:22 fastvm-rhel-8-0-23 pacemaker-execd [1211060] (log_execute) debug: executing - rsc:vmfence action:monitor call_id:35
Jul 16 12:30:22 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (schedule_stonith_command) debug: Scheduling 'monitor' action on vmfence for a1bcd338-f627-44d3-a9a7-90bce44756bb (timeout=30s)
Jul 16 12:30:22 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (fork_cb) debug: Operation 'monitor' on vmfence now running with pid=1211454, timeout=30s
...
Jul 16 12:30:52 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (child_timeout_callback) warning: fence_vmware_soap_monitor_1 process (PID 1211454) timed out
Jul 16 12:30:52 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (operation_finished) warning: fence_vmware_soap_monitor_1[1211454] timed out after 30000ms
Jul 16 12:30:52 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (st_child_done) debug: Operation 'monitor' on 'vmfence' completed with rc=-62 (0 remaining)
Jul 16 12:30:52 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (log_operation) notice: Operation 'monitor' [1211454] for device 'vmfence' returned: -62 (Timer expired)
# without pcmk_monitor_timeout
Resource: vmfence (class=stonith type=fence_vmware_soap)
Attributes: <opts> pcmk_monitor_retries=1 pcmk_reboot_retries=1 ssl_insecure=1
Jul 16 12:27:00 fastvm-rhel-8-0-23 pacemaker-execd [1211060] (log_execute) debug: executing - rsc:vmfence action:monitor call_id:20
Jul 16 12:27:00 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (schedule_stonith_command) debug: Scheduling 'monitor' action on vmfence for a1bcd338-f627-44d3-a9a7-90bce44756bb (timeout=20s)
Jul 16 12:27:00 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (fork_cb) debug: Operation 'monitor' on vmfence now running with pid=1211366, timeout=20s
...
Jul 16 12:27:20 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (child_timeout_callback) warning: fence_vmware_soap_monitor_1 process (PID 1211366) timed out
Jul 16 12:27:20 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (operation_finished) warning: fence_vmware_soap_monitor_1[1211366] timed out after 20000ms
Jul 16 12:27:20 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (log_operation) notice: Operation 'monitor' [1211366] for device 'vmfence' returned: -62 (Timer expired)
Jul 16 12:27:20 fastvm-rhel-8-0-23 pacemaker-execd [1211060] (log_finished) debug: vmfence monitor (call 20) exited with status 198 (execution time 20007ms, queue time 0ms)
# without pcmk_monitor_timeout, and with CRM_meta_timeout set explicitly to 40s
Resource: vmfence (class=stonith type=fence_vmware_soap)
Attributes: <opts> pcmk_monitor_retries=1 pcmk_reboot_retries=1 ssl_insecure=1
Operations: monitor interval=60s timeout=40s (vmfence-monitor-interval-60s)
Jul 16 12:36:50 fastvm-rhel-8-0-23 pacemaker-execd [1211060] (log_execute) debug: executing - rsc:vmfence action:monitor call_id:48
Jul 16 12:36:50 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (schedule_stonith_command) debug: Scheduling 'monitor' action on vmfence for a1bcd338-f627-44d3-a9a7-90bce44756bb (timeout=40s)
Jul 16 12:36:50 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (fork_cb) debug: Operation 'monitor' on vmfence now running with pid=1211526, timeout=40s
...
Jul 16 12:37:30 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (child_timeout_callback) warning: fence_vmware_soap_monitor_1 process (PID 1211526) timed out
Jul 16 12:37:30 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (operation_finished) warning: fence_vmware_soap_monitor_1[1211526] timed out after 40000ms
Jul 16 12:37:30 fastvm-rhel-8-0-23 pacemaker-fenced [1211059] (log_operation) notice: Operation 'monitor' [1211526] for device 'vmfence' returned: -62 (Timer expired)
Jul 16 12:37:30 fastvm-rhel-8-0-23 pacemaker-execd [1211060] (log_finished) debug: vmfence monitor (call 48) exited with status 198 (execution time 40006ms, queue time 0ms)
Start operations are a bit of a different story (see #2108)
How to document all that without driving the user mad is an open question ;)
Given all these complications, it may be best not to specify a default in the doc. Rather, we could say that the default is determined dynamically by a number of factors, and that specifying these pcmk_<action>_timeout attributes will override the defaults and set the timeouts explicitly.
Pacemaker itself does not use or set this at all, but I believe some Linux-HA-style fencing agents (which use fence_legacy) use it.
Maybe. I've never personally encountered a fence agent that uses stonith-timeout, and I can't find any reference to it in the current fence-agents upstream besides an incorrect example command: https://github.com/ClusterLabs/fence-agents/blob/master/agents/rcd_serial/fence_rcd_serial.py#L12
Linux HA style fence agents pre-date pacemaker and are supported on some distros via the cluster-glue package. In pacemaker such agents have to use the fence_legacy wrapper to map their interface to the RHCS style that pacemaker natively uses. I've never used them myself so I'm unclear how they might interact with stonith-timeout.
I had tested list and status with stonith_admin previously. Now, with internally scheduled list and status commands, I'm seeing 60s.
I'm guessing this 60s comes from schedule_internal_command(), so there's always a timeout set by the time it gets to get_capable_devices() and it never uses DEFAULT_QUERY_TIMEOUT. But I didn't trace through to see if there's some code path that gets there without that.
Given all these complications, it may be best not to specify a default in the doc. Rather, we could say that the default is determined dynamically by a number of factors, and that specifying these
pcmk_<action>_timeoutattributes will override the defaults and set the timeouts explicitly.
Probably best for everyone involved. :)
We do still need to update the stonith-timeout documentation, something like:
Default timeout for cluster-initiated fencing actions
When the cluster (as opposed to the administrator via stonith_admin, or external software) initiates a fencing action (such as "off" or "reboot") against a node, this will be the initial timeout value used for the action if a more specific value such as pcmk_reboot_timeout has not been set for any device used to fence. However, the fencer may dynamically adjust the timeout after the request is initiated.