lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

S3 blockstore Delete uses wait to verify object is gone

Open nopcoder opened this issue 2 years ago • 5 comments

S3 should enable read after write - no need to wait for object to be deleted. This makes the call expensive and slow.

nopcoder avatar Oct 22 '23 14:10 nopcoder

I know that I asked for this :-) But I'm having second thoughts. It probably needs to be optional because not all blockstores will be suitable. Maybe even default false.

While S3 is consistent like this, many other S3-"compatible" blockstores will not be compatible. For instance, MinIO on some NFS-mounted drives might not be consistent, or on Ext4. See this MinIO blog for how hard this can be. That blog also hints that some unnamed blockstores might not have {list,read}-after-write consistency.

arielshaqed avatar Oct 23 '23 06:10 arielshaqed

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

github-actions[bot] avatar Jan 22 '24 01:01 github-actions[bot]

Closing this issue because it has been stale for 7 days with no activity.

github-actions[bot] avatar Jan 30 '24 01:01 github-actions[bot]

Re-opening: there is potential for some modest performance gain here!

arielshaqed avatar Jan 30 '24 05:01 arielshaqed

Re-opening: there is potential for some modest performance gain here!

@arielshaqed Care to explain? Why do we even wait in the first place?

itaiad200 avatar Mar 19 '25 13:03 itaiad200

In practice we no longer use Adapter.Remove. Closing this, in favour of removing Remove.

arielshaqed avatar Oct 28 '25 15:10 arielshaqed