Support for persisted deletes (or blocklists)

[frodelu]

There are use-cases - especially with regards to personal data - where you want to delete a document and ensure that it stays deleted regardless of another PUT against the same docid at a later time. This can typically happen when you have multiple asynchronous processing pipelines, e.g. grid for batch processing and a real-time pipeline for updates from the UI.

One option is to support a dedicated (permanent) blacklist mechanism, but my initial guess is that for this feature it would be sufficient to re-use the remove-entries we already keep in proton. So if a document comes in and it already has a "persistent" (flag per delete) remove-entry, it should be discarded. This way, the application owner can tune the length (and thus cost) of this persistency with the pruneremoveddocumentsage setting or, possibly, a new separate age setting for "persistent" remove-entries.

With GDPR looming, I am guessing this future would be of increased interest going forward.