Towards first release

[tomjridge]

This issue tracks the various things I need to do before first release.

• [x] Add RO processes
• [x] add logging as is done in mirage/index and other mirage libs
• [x] take care with corrupted files : logs (logs - have size field in file header)
• [x] take care with corrupted files: partition - rename trick, freelist - rename trick
• [ ] log_file: perhaps allow reading beyond ppos up to end of file, for RO instance
• [ ] integrate log_file with frontend and frontend_ro
• [ ] make most errors explicit in return type, document other exceptions
• [ ] simplify construction of various objects
• [ ] use a single directory rather than have lots of different config filenames
• [ ] simplify construction of objects - pass subcomponents not filenames etc
• [ ] cope with possibility of hash collisions?
• [ ] consider supporting delete
• [ ] remove old files (eg old log file, after merge)
• [ ] Test RO processes
• [ ] Don't cache the buckets file by default (?) - or perhaps switch to not caching when memory is tight (but how to detect?)
• [ ] (minor) change default ~fn args to match config names
• [ ] check basic functionality: can open an existing store etc
• [ ] add syncs in appropriate places
• [ ] some performance figures
• [ ] user documentation - how to get started etc; maybe a few webpages
• [ ] ocamlformat, tidy code and ocamldoc; add more ocamldoc
• [ ] tidy bucket code
• [ ] fixup all FIXMEs in the code
• [ ] add error checking to create/open functions (or at least be sure what exceptions can occur
• [ ] add timing on external (public API, disk intf) intfs
• [ ] fixup documentation for bucket reuse

For Irmin team:

• [ ] explain the various ways this improves on the previous index eg simplicity; no locking; low memory usage; performance; resistance to FS behaviours (write reordering etc); behaviour on crash; etc

For Tezos:

• [ ] integrate with Tezos node, so that all the RO processes get created correctly, etc.

Maybe:

• [ ] Add mmap implementation for partition?
• [ ] Fuse irmin-pack with kv-hash
• [ ] Some support for hot/cold data ("layered store")
• [ ] support for mirage? (we don't need much - just to implement the various individual files on a block device)