lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Use previousRunID to accelerate GC jobs

Open talSofer opened this issue 2 years ago • 0 comments

Address the TODO in https://github.com/treeverse/lakeFS/blob/5b7547c5606120bb1869bc5ab03eecd2555e1157/clients/spark/core/src/main/scala/io/treeverse/clients/GarbageCollector.scala#L263

This issue requires to understand how previousRunID is currently used by GC, and talk to @guy-har to understand the problems and potential solutions.

See performance improvement potential analysis by @arielshaqed in https://github.com/treeverse/cloud-controlplane/issues/414#issuecomment-1195431320

talSofer avatar Apr 03 '22 05:04 talSofer