Add trim and scrub commands for maintaining a ZFS storage in EVE

[vk-en]

This PR adds an implementation to EVE that will allow TRIM and Scrub commands to be received from the controller, with the ability to run them on a specific schedule.

A proposal for adding this functionality can be found here.

[zedi-pramodh]

I think the trim and scrub commands should be internal to EVE/ZFS and should not be exposed through API (at least to start with).

I am thinking the implementation should be something like a scheduler with calls zpool trim and zpool scrub commands at periodic intervals. Since EVE does not support systemd, we probably are better off implementing a job scheduler within EVE and make it execute the trim and scrub commands.

[vk-en]

@zedi-pramodh I believe that we can only have 2 options for executing these commands.

1. auto-TRIM in ZFS We can set the TRIM auto-execution property with the autotrim option. This will allow us not to worry about TRIM in our case and be fully automated. But for the Scrub command, there is no such option.
2. TRIM/Scrub only on request from the controller or run according to a schedule set through the controller. This allows us to be much more flexible than just functionality that periodically takes up resources on its own. I think we still want to provide something universal. I understand that we should make sure that there are as few settings as possible. But I don't think we can let it work independently for several reasons:

• These operations are resource-intensive in their execution and will negatively affect the performance of applications and virtual machines, at least for read operations.
• They can take an indefinite amount of time if the disks are large.
• EVE is a universal tool, and everyone's needs are different, it may not be useful TRIM/Scrub to someone at all.

I believe it should always be possible to change the schedule/start/disable the execution of these commands on the controller.

I do not insist, this is just my opinion on the situation with these commands. Let's continue the discussion here, I think that sooner or later we should come to some kind of decision on this issue.

[rouming]

Please, no. We don't expose low-level API to the zedcloud. Especially filesystem API. No. @eriknordmark @zedi-pramodh let's close this.

[zedi-pramodh]

Nothing wrong with this PR. We need to run TRIM and SCRUB on the EVE devices running zfs once a month, that's the recommendation from our third party company. There are two ways to do that, we can automatically schedule in ZFS or ask user to pick and choose their schedule.

I believe this PR is for user to choose the schedule.

Configuring automatically in ZFS is a good option but how can we choose the schedule for the customers ? Lets discuss and pick the best option.

[rouming]

There are two ways to do that, we can automatically schedule in ZFS or ask user to pick and choose their schedule.

Great, zedcontroller sends a configuration option, EVE acts as a cron daemon and does a scheduling of any internal operations according to the backend FS. Why we expose a filesystem API for that or I missed the whole point?

[rouming]

Configuring automatically in ZFS is a good option but how can we choose the schedule for the customers ?

Simplified cron-like configuration string in the config?

[zedi-pramodh]

There are two ways to do that, we can automatically schedule in ZFS or ask user to pick and choose their schedule.

Great, zedcontroller sends a configuration option, EVE acts as a cron daemon and does a scheduling of any internal operations according to the backend FS. Why we expose a filesystem API for that or I missed the whole point?

Does cron job survive the EVE upgrades ? We do not have cron mechanism so far in EVE.

IMO we have to take one of the following approaches:

1. via API (this PR)

2. build a generic framework (micro service) to run some periodic tasks in EVE. Then that service can be given the scrub/trim task. Even then the scheduling part need to be told to that somehow.

3. Have a config parameter for EVE with trim.schedule=monthly or something like that. EVE picks it up launches a thread and runs in a loop for ever. Problem is if device reboots then schedule gets started again (may be OK)

option#1 is good (this PR) but that requires changes in controller which I agree with you that it does not make sense for customer to know about these storage internal housekeeping.

option#3 seems self contained.

[rouming]

There are two ways to do that, we can automatically schedule in ZFS or ask user to pick and choose their schedule.

Great, zedcontroller sends a configuration option, EVE acts as a cron daemon and does a scheduling of any internal operations according to the backend FS. Why we expose a filesystem API for that or I missed the whole point?

Does cron job survive the EVE upgrades ? We do not have cron mechanism so far in EVE.

Since we have a persistent node config, crond can be configured on each reboot, then trim/scrub can be called as an independent tools, which is also nice. So reboots should not be a problem.

Also scheduling can be done by means of the eve (pillar) itself, which should be easy to do (it seems option#3 you mentioned above).

IMO we have to take one of the following approaches:

1. via API (this PR)

Please, no :) We don't expose low-level filesystem API.

2. build a generic framework (micro service) to run some periodic tasks in EVE. Then that service can be given the scrub/trim task. Even then the scheduling part need to be told to that somehow.
3. Have a config parameter for EVE with trim.schedule=monthly or something like that. EVE picks it up launches a thread and runs in a loop for ever. Problem is if device reboots then schedule gets started again (may be OK)

For example cron-like configuration format (or simpler): trim.schedule="0 0 1 1-12 1"

“At 00:00 on day-of-month 1 and on Monday in every month from January through December.”

[eriknordmark]

To me a key question is whether the average or power user cares about when these are scheduled. If not we can have a default plus a somewhat hard to use trim.schedule="0 0 1 1-12 1" configItem.

But if they don't want these to run during performance critical work on the device, then we might need to make this be more user friendly to configure, maybe in the controller UI and API. That doesn't necessarily need to expose the details of the filesystem but could be a generic "storage maintence task schedule". But it is probably premature to do that; with ZFS I assume we also need to run periodic scrubs, which might have more of a perfornance impact. So a trim.schedule configItem seems like a reasonable start.

[rouming]

On UI configuration can be very friendly of course: No trim, Daily trim, Weekly trim, Monthly trim. But eve itself can support more complicated configuration string, similar to trim.schedule="0 0 1 1-12 1"

Scrubs can have similar configuration format: scrub.schedule="0 0 1 1-12 1", UI shows an option if underlying fs supports trim/scrub.

[zedi-pramodh]

So a trim.schedule configItem seems like a reasonable start.

Yes basically the recommendation is to run scrub and trim every month and both not at same time. That will impact performance. So something like, scrub on first of month at midnight and trim on 7th of every month at midnight.

We can default the config parameters to that and if user knows what they are doing and wants to modify its up to them.

[rouming]

@zedi-pramodh do we have anyone to continue this work or we can close and reopen once needed?

[zedi-pramodh]

I already filed an internal ticket to fix this issue using EVE config as discussed above. We can close this PR.