S3 backup issue tracker

[marcocastignoli]

We discussed in the meeting today about the tasks left about the S3 and Filebase IPFS sync. This is an issue to track the tasks regarding this.

• [ ] Setup IPNS update task on GCP that has to run after aws s3 completes
  • There is no straightforward option to trigger another Cloud Run Job / Cloud Function after a Cloud Run jobs finishes. I found a solution that uses LogRouter to forward sourcify-staging-s3-sync logs (filtered by "has completed successfully", probably there is a better filter) to a PubSub topic. Every time this PubSub topic receives a message a Cloud Function is executed.
  • [ ] Remember to delete GPC test entities: LogRouter,Cloud Function,PubSubTopic
• [ ] The "Generate CID" button was not working on Filebase. Check with Filebase team about this. We assume this is the root CID of the repo. Eventually we need to have the root CID to publish this as the IPNS value.
  • ~~I contacted Filebase support, waiting for a response~~
  • They told me that there is a limit for buckets with max 25k files in order to generate the CID, I asked if they can do something specifically for us; waiting for a response
• [ ] Setup regular s3 syncs with schedule
  • [x] Setup staging schedule
• [x] Do some cleaning on the GCP jobs (rename, delete)
• [x] Fix the s3-backup-check test on CircleCI running daily. Since we don't use the Amazon S3 bucket as backup anymore, we should point this check to the Filebase S3. https://github.com/ethereum/sourcify/pull/1580