Proposal: lakeFS hard-delete

[N-o-Z]

Closes #1933 (but not really...)

Design proposals document for in-house hard-delete in lakeFS

The

[N-o-Z]

More or less hit a brick wall with this 😅 Please see the summary for all the proposal attempts, please share if you have any ideas

[N-o-Z]

Thank you @N-o-Z for uploading suggestion #4 As the general vibe is that this (#4) is the solution we are currently aiming at, can you please add a bit more details? Make it more workable, towards implementation? Thanks

@itai-david Thank you for the very valuable input. Please see the updated proposal and please let me know how else can be improved

[N-o-Z]

Thanks!

IIUC we've ruled out options 1-4. Can we perhaps get rid of them? It would make it easier (for me) to focus. mag

Thanks @arielshaqed I feel that as long as we didn't settle on any proposal, we should keep all of the proposals information. In case this will be eventually rejected we will benefit from having documentation of all the attempted solutions for future reference.

[wengchenyang1]

Thanks! IIUC we've ruled out options 1-4. Can we perhaps get rid of them? It would make it easier (for me) to focus. mag

Thanks @arielshaqed I feel that as long as we didn't settle on any proposal, we should keep all of the proposals information. In case this will be eventually rejected we will benefit from having documentation of all the attempted solutions for future reference.

Hello @N-o-Z , may I ask for the progress of proposal 5? We face compliance issue because some files are not hard deleted on s3.

[N-o-Z]

@wengchenyang1, All the suggested proposals in this document have inherent faults which make non of them a viable solution. We've decided for the time being to work in other directions for this problem. We''re currently working on designing an offline solution similar to the current GC process we have. I'll make sure to tag you once the PR is opened

[itaiad200]

@N-o-Z can we merge this proposal to rejected?

[N-o-Z]

@N-o-Z can we merge this proposal to rejected?

Yes - lets do that

[N-o-Z]

Although this was rejected, the hard work on this suggestion proved us that an online consistent, atomic & maintainable solution to this problem is not feasible. This output is better than implementing a design that cannot work :)

@itaiad200, I still want to believe that in the future we will be able to find a working solution for this problem