Scaling a journal

[philrz]

The following text was present in a retired "lake design" document (see #3803). It has been established that this was really a pending to-do, so this issue tracks its ultimate implementation and corresponding updates to docs.

#### Scaling a Journal

When the sizes of the journal snapshot files exceed a certain size
(and thus becomes too large to conveniently handle in memory),
the snapshots can be converted to and stored
in an internal sub-pool called the "snapshot pool".  The snapshot pool's
pool key is the "from" value (of its parent pool key) from each commit action.
In this case, commits to the parent pool are made in the same fashion,
but instead of snapshotting updates into a snapshot ZNG file,
the snapshots are committed to the journal sub-pool.  In this way, commit histories
can be rolled up and organized by the pool key.  Likewise, retention policies
based on the pool key can remove not just data objects from the main pool but
also data objects in the journal pool comprising committed data that falls
outside of the retention boundary.

> Note we currently record a delete using only the object ID.  In order to
> organize add and delete actions around key spans, we need to add the span
> metadata to the delete action just as it exists in the add action.

@mccanne also once offered the following related thought in duplicate issue #2787:

we shouldn’t need this for a while as this is only needed at truly massive cloud scale. We might want to put thought into it sooner rather than later because we need an API endpoint to scan the log optionally with a Zed query so clients can do intelligent orchestration even before we get to scale. This may or may not be the current list endpoint.