cleanup: reconsider options for redirecting/cleaning up sqlite

[garlick]

trafficstars

Large ensemble runs using flux have left many ~20GB sqlite files in /tmp over time on one of our large systems.

Currently, flux creates the sqlite file on rank 0 in a per-instance directory in /tmp, and cleans it up on exit.

One can redirect it to another location with the persist-filesystem or persist-directory attribute settings, but this disables cleanup.

We should think carefully about the way people are using single-user flux and how we want to handle these large files. At minimum it seems like we should have separate options for redirecting and disabling cleanup of the sqlite file.

Also, we should make sure we have test coverage for cleanup (or persistence).

[garlick]

The above persist-* attributes are no more. Now one just sets statedir to both set the location of persistent state, and to imply that persistent state is required. We still don't have a way to relocate the sqlite file and not keep it.

See also: FAQ entry and flux-broker-attributes(7).