flux-core icon indicating copy to clipboard operation
flux-core copied to clipboard

WIP content-sqlite: preallocate space to avoid outside diskfull events

Open chu11 opened this issue 6 months ago • 9 comments

Per discussion in #6169

Support a new "preallocate" config & module option to pre-allocate a specific amount of space to sqlite so that we can hopefully avoid ENOSPC issues when outside actors fill up the disk. Option name debatable ... "preallocate_size"?

The technique is to basically create a database table, write a bunch a data to it, then drop the table. This won't work with journaling b/c the journal needs space too. But code is added to disable journaling if ENOSPC is hit.

There's a lot of "setup" commits in here, we could probably split out into another PR.

TODOs

  • [x] ~~option take meg/gig suffix?~~ ehhh scratch that, i don't what the config file inputs to always be strings
  • [x] test at bigger scale/size (i.e. not 5m mounts)
  • [x] I probably should document this somewhere
  • [ ] BIG THING TO DISCUSS IN ISSUE - how to put database back into journaling mode when space issues fixed. Disabling journaling does lead to a healthy slowdown in performance (job throughupt maybe 15% down). This can probably be a different issue/PR, but needs to be discussed.

chu11 avatar Aug 15 '24 18:08 chu11