zed icon indicating copy to clipboard operation
zed copied to clipboard

Vacuum support

Open philrz opened this issue 3 years ago • 0 comments

The following text was present in a retired "lake design" document (see #3803). It has been established that this was really a pending to-do, so this issue tracks its ultimate implementation and corresponding updates to docs.

## Vacuum Support

While data objects currently can be deleted from a lake, the underlying data
is retained to support time travel.

The system must also support purging of old data so that retention policies
can be implemented.

This could be supported with the DANGER-ZONE command `zed vacuum`
(implementation tracked in [zed/2545](https://github.com/brimdata/zed/issues/2545)).
The commits still appear in the log but scans at any time-travel point
where the commit is present will fail to scan the deleted data.
In this case, perhaps we should emit a structured Zed error describing
the meta-data of the object that was unavailable.

Alternatively, old data can be removed from the system using a safer
command (but still in the DANGER-ZONE), `zed vacate` (also
[zed/2545](https://github.com/brimdata/zed/issues/2545)) which moves
the tail of the commit journal forward and removes any data no longer
accessible through the modified commit journal.

philrz avatar Sep 27 '22 23:09 philrz