End-to-end test s3 backups

[kuzdogan]

The contract repository is currently being synced to an Amazon S3 bucket as a backup daily. We should also test if these backups are succeeded in nightly tests and get notified if it fails.

One way to do this is to make use of the previous day's end-to-end tests, which deploy contracts to testnets. The e2e-s3 test of the current day could check if the contracts deployed the previous day were successfully backed up in the S3 bucked.

Say on the previous day the contract 0x29F32887d9426b37b8758afa3723a8e2790D83ed was deployed to Goerli (5). Then, the e2e-s3 test of the current day should check if s3://sourcify-backup-s3/stable/repository/contracts/full_match/5/0x29F32887d9426b37b8758afa3723a8e2790D83ed/metadata.json exists. If so, it indicates the contracts from the previous day are processed by the s3 backup container.

One caveat is that sometimes the e2e tests sometimes fail to deploy contracts. For instance, if the previous day's e2e goerli test was not able to deploy a contract for some reason (gas, network etc.), the e2e-s3 test of the current day will also fail. As a workaround contracts from all of the testnets can be checked, and if one of them is found on the S3 bucket, it would indicate the contracts are backed up. It is highly unlikely that all 3 testnet deployments fail at the same time (if so, probably there's a bigger problem with verification). Additionally, the deployed contracts from e2e-monitoring tests can be checked as well which in turn makes total 3+3=6 contract checks for backup. If any of these passes, the S3-test passes.

[marcocastignoli]

This is how we are going to implement this new feature.

1. Edit scripts/verification-e2e.js so it saves the verified contract in a folder and then add to .circleci/nightly.yml (in the verification-e2e-base alias) the store_artifacts to save the verified contract in the folder

- store_artifacts:
          path: ./verified-contracts

2. create a new job in the nightly that takes the artifact from the verification-e2e job and uses it to check the existance in the sourcify-backup-s3 bucket

[marcocastignoli]

This commit contains what described above: https://github.com/ethereum/sourcify/commit/871bfac6cbeb1cbd40335f8feb239e428650885a

[kuzdogan]

Reopening, close once confirmed on CircleCI

[marcocastignoli]

Confirmed by CircleCI