Al Chu

> Could we cut this PR there and save the discussion about making it the default for another time? ahhh, I missed that. I'll tweak the PR. Should be simpler...

re-pushed - split out content checkpoint code into another file - the "context' that the `content-checkpoint` API creates is actually stored within `struct content_cache` and not `struct broker`. I didn't...

> Edit: just realized this checkpoint service is created on all ranks, but I think it should only be created on rank 0? If for some reason it is accessed...

re-pushed, addressing all of the comments above except the discussion about if `content.flush` should be an error or not. Of particular note: - created a bunch of new functions to...

> It might be a good idea to pause and question whether the "flush" operation should fail when there is no backing store. Maybe we should redefine it as "flush...

argh, re-pushed, fixed up a mem-leak and fixed some bash-isms in my tests that were affecting the CI

and re-pushed, fixing a s3 CI issue

re-pushed, building on top of #4519, adjust some tests as a result of #4519

rebased and re-pushed now that #4519 is merged, this PR is a lot smaller now :-)

re-started one builder that had a ton of failures not related to this PR. assumption is workflow/container borked and affected bunch of tests.