neon icon indicating copy to clipboard operation
neon copied to clipboard

Fix partial upload bug with invalid remote state

Open petuhovskiy opened this issue 7 months ago • 3 comments

We have an issue that some partial uploaded segments can be actually missing in remote storage. I found this issue when was looking at the logs in staging, and it can be triggered by failed uploads:

  1. Code tries to upload SEG_TERM_LSN_LSN_sk5.partial, but receives error from S3
  2. The failed attempt is saved to segments vec
  3. After some time, the code tries to upload SEG_TERM_LSN_LSN_sk5.partial again
  4. This time the upload is successful and code calls gc() to delete previous uploads
  5. Since new object and old object share the same name, uploaded data gets deleted from remote storage

This commit fixes the issue by patching gc() not to delete objects with the same name as currently uploaded.

petuhovskiy avatar Jul 15 '24 14:07 petuhovskiy