neon
neon copied to clipboard
Fix partial upload bug with invalid remote state
We have an issue that some partial uploaded segments can be actually missing in remote storage. I found this issue when was looking at the logs in staging, and it can be triggered by failed uploads:
- Code tries to upload
SEG_TERM_LSN_LSN_sk5.partial
, but receives error from S3 - The failed attempt is saved to
segments
vec - After some time, the code tries to upload
SEG_TERM_LSN_LSN_sk5.partial
again - This time the upload is successful and code calls
gc()
to delete previous uploads - Since new object and old object share the same name, uploaded data gets deleted from remote storage
This commit fixes the issue by patching gc()
not to delete objects with the same name as currently uploaded.