zed: Add synchronous zedlets

[tonyhutter]

Motivation and Context

Allow zedlets to execute synchronously.

Description

Historically, ZED has blindly spawned off zedlets in parallel and never worried about their completion order. This means that you can potentially have zedlets for event number 2 starting before zedlets for event number 1 had finished. Most of the time this is fine, and it actually helps a lot when the system is getting spammed with hundreds of events.

However, there are times when you want your zedlets to be executed in sequence with the event ID. That is where synchronous zedlets come in.

ZED will wait for all previously spawned zedlets to finish before running a synchronous zedlet. Synchronous zedlets are guaranteed to be the only zedlet running. No other zedlets may run in parallel with a synchronous zedlet. Users should be careful to only use synchronous zedlets when needed, since they decrease parallelism.

To make a zedlet synchronous, simply add a "-sync-" immediately following the event name in the zedlet's file name:

	EVENT_NAME-sync-ZEDLETNAME.sh

For example, if you wanted a synchronous statechange script:

	statechange-sync-myzedlet.sh

How Has This Been Tested?

Test case added

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Performance enhancement (non-breaking change which improves efficiency)
• [ ] Code cleanup (non-breaking change which makes code smaller or more readable)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• [ ] Documentation (a change to man pages or other documentation)

Checklist:

• [ ] My code follows the OpenZFS code style requirements.
• [ ] I have updated the documentation accordingly.
• [ ] I have read the contributing document.
• [ ] I have added tests to cover my changes.
• [ ] I have run the ZFS Test Suite with this change applied.
• [ ] All commit messages are properly formatted and contain Signed-off-by.

[tonyhutter]

@behlendorf I updated this with your changes, including a scrub_finish check.

[tonyhutter]

Are there any existing zedlets we ship which should be made synchronous as part of this change?

Probably these:

deadman-slot_off.sh
statechange-slot_off.sh

...potentially the LED scripts too:

statechange-led.sh
vdev_clear-led.sh
vdev_attach-led.sh
pool_import-led.sh

However, the LED scripts can take in the milliseconds to seconds range to change the LED value, so I can understand keeping them async.

[tonyhutter]

I ended up marking all those zedlets I mentioned earlier as synchronous in my latest push.

[tonyhutter]

It was mentioned at the OpenZFS meeting today that we should add in timeouts for the synchronous zedlets. Let me look into that...

[tonyhutter]

I've begun to have second thoughts about adding a zedlet timeout:

• Zedlets have existed for 11 years without timeouts

• What action do we take on a timeout? Kill the script? Couldn't that be dangerous (like if the zedlet is flashing firmware)?

• If we did implement timeouts, would we do it for both sync and async zedlets? Or just sync?

• Different zedlets would want different timeout values. How would you wire these per-zedlet timeout values into zed?

• Users can can already (manually) put a timeout in their zedlets if they want:

#!/bin/bash
(sleep $TIMEOUT && kill "$$") & 
...
< zedlet lines >
...
kill $(jobs -p)

[tonyhutter]

I re-based this yesterday. Should be good to go.