Existing PSA_Requests should be available to Elastic Provider

[flea89]

Overarching user needs / context:

• When migrating pinning to Elastic Provider (pick-up service), we will need the existing pinned pins to be on S3 so the Elastic Provider can provide them.
• Also, we need to make sure they are stored in 2 systems rather than 1, so we will still be able to retrieve pinned files from IPFS once we migrate the PSA to use picup.

Scope of this ticket:

• A backup job that copies existing pins that are marked as pinned to S3. It should be written in a way that it can be run repeatedly from cron, to keep backing up pinned pins.

Acceptance criteria:

• All pinned pins, processed through PSA to cluster should appear in S3
• The script/job can be run multiple times

[mbommerez]

Removing this from the PSA Epic and adding as a separate item to the backlog.

[mbommerez]

Discussed with @alanshaw @flea89 @francois-potato .

• Existing CRON job could be repurposed.
• We cannot back up pins straight away, as we don't have the data.
• As soon as we get the pin request we could fetch it in the background and send it to S3.
• Makes sense to put it in Elastic Provider.
• Regardless of the above, we should be backing up files we have pinned, as we do for uploads.
• It needs to be built in a way that it can be used in NFT.storage as well

[olizilla]

We should implement a backup job that copies existing pins that are marked as pinned to S3. It should be written in a way that it can be run repeatedly from cron, to keep backing up pinned pins. We need to unblock that part of the work asap. Is anything else needed here @flea89 @mbommerez ?

[flea89]

@alanshaw wrote something back in the days, I can definitively have a look and re-purpose/adapt if needed.

@olizilla I'd like to understand how this relates to Elastic Provider work.... I'm wondering if, with the rise of EP, this task can be translated to "let's make existing PSA_REQUESTS available to EP, rather than back up pins from cluster to a "cold storage". Is this the right assumption? Or that will come after?

[olizilla]

We need the existing pinned pins to be on S3 so the elastic provider can provide them, and so that they are stored in 2 systems rather than 1. So yes, you can reframe it as let's make existing PSA_REQUESTS available to EP if that helps.

[flea89]

Yup, that makes sense, thanks for validating my assumptions!

That been clarified, the reason for blocking it was to understand with you if this had any dependency with your service. Given atm we don't have a huge amount of context on how you're approaching it and we haven't discussed yet how w3 and service will integrate, I was keen to get your POV before approaching this.

Anyway with the assumption psa state is kept in w3 (not in your service) it shouldn't be a big deal. But please advise if I'm missing anything. I'll unblock.

[mbommerez]

Repurposing this ticket as a migration script ticket.

[mbommerez]

• Most issues have been fixed, this is close to be done.
• New end-2-end tests have been added (try to reach cluster) on top of the previous mocked ones.
• We cannot reach the cluster peer locally with Dagula - this could be to do with cluster setup with docker, rather than a code problem.