lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Add list of potential wins for LakeFSOutputCommitter

Open arielshaqed opened this issue 1 year ago • 1 comments

Fixes #4064.

arielshaqed avatar Aug 31 '22 14:08 arielshaqed

Thanks @arielshaqed .

Some pain points related to S3 that I wish this PR will resolve are:

  1. S3 Consistency used to be a pain and was solved with strong read after write for individual files which made EMRFS obsolete. but lack of atomic directories operations is a pain – copying a large set of parquet files (10k) and having multiple readers and writers at the same time can be a challenge when looking at the directories/repositories as a whole and not a file-by-file consistency. Hence, atomic operations on an s3 directory are desired for working with intense data applications and will solve many inconsistency issues with data. Another example - Having many readers to table with 1K parquet files and one update – update is not an atomic action, and readers might read the wrong data.

more issues with S3, that could be nice to resolve are:

  1. AWS will often return HTTP 503 Slow Down errors for request rates that are much, much lower than their advertised limits.
  2. You are supposed to code clients with backoffs/retries adequate to absorb arbitrary levels of HTTP 503 Slow Down until they finish scaling (which is entirely unobservable and could be minutes, hours, or days).

I understand that this PR will not solve 2+3, but it has the potential to drastically reduce metadata ops during writes which can improve the experience and number of errors or "slow down" exceptions.

Thank you again for looking into it.

adipolak avatar Sep 08 '22 22:09 adipolak