WIP: broker: add ability to mmap files into content cache

[garlick]

I'm parking this weekend experiment pending some time to test scalability.

This adds a feature I had been thinking about for some time - support for mmapping a file into the broker content cache on rank 0, and reading it through the cache on the other broker ranks of the instance, leveraging the tree based overlay network fanout for scalability.

For preliminary tooling I added

• flux content mmap FILENAME [BLOBSIZE] which maps a file and lists blobrefs for chunks on stdout
• flux content mumnap FILE which unmaps the file
• a new flux content load mode which reads blobrefs on stdin and contcatenates the blobs on stdout

With this one can do something like:

ƒ(s=64,d=0,builddir) $ flux content mmap 4g.img | flux exec sh -c "flux content load | md5sum"
[snip]
ƒ(s=64,d=0,builddir) $ flux content munmap 4g.img

In that command, the list of blobrefs is being distributed to the stdin of the remote commands through flux exec, then the file contents is being loaded through the cache.

On the relevance to the file broadcast tool being discussed in #3744 and #3631, I'm not sure if this is a direct fit because

• this service is only available to the instance owner
• the source file must be on broker rank 0
• the source must be a file (it cannot be a stream)

If we did want to go forward with such a tool though, I was thinking it could be made standalone so it could map the file, perform the copy by using the broker subprocess exec service directly, and then unmap the file when done. In the above example, the file has to be manually unmapped after the command completes.

Mostly though, in this PR, I just wanted to see if we could implement the basic capability since it's been on the back burner for so long, and then we'll at least have something to compare other ideas to. Sometimes these demons must be exorcised.

[codecov[bot]]

Codecov Report

Merging #4546 (abfac93) into master (787df67) will decrease coverage by 0.02%. The diff coverage is 84.52%.

:exclamation: Current head abfac93 differs from pull request most recent head ccbf65f. Consider uploading reports for the commit ccbf65f to get more accurate results

@@            Coverage Diff             @@
##           master    #4546      +/-   ##
==========================================
- Coverage   83.39%   83.36%   -0.03%     
==========================================
  Files         403      404       +1     
  Lines       68040    68311     +271     
==========================================
+ Hits        56740    56949     +209     
- Misses      11300    11362      +62     

Impacted Files	Coverage Δ
src/broker/content-checkpoint.c	77.43% <ø> (ø)
src/cmd/builtin/content.c	78.70% <79.74%> (+0.44%)	:arrow_up:
src/broker/content-mmap.c	81.36% <81.36%> (ø)
src/common/libflux/message.c	93.05% <90.00%> (-0.04%)	:arrow_down:
src/broker/content-cache.c	86.70% <95.89%> (+1.05%)	:arrow_up:
src/modules/job-exec/bulk-exec.c	79.35% <0.00%> (-1.48%)	:arrow_down:
src/modules/job-info/guest_watch.c	76.75% <0.00%> (-0.82%)	:arrow_down:
src/modules/job-archive/job-archive.c	62.62% <0.00%> (-0.70%)	:arrow_down:
src/broker/overlay.c	85.86% <0.00%> (-0.64%)	:arrow_down:
src/common/libsubprocess/local.c	83.90% <0.00%> (-0.49%)	:arrow_down:
... and 3 more

[garlick]

Reworked PR coming soon, closing this one.