Download URL for Import deals

[anjor]

As a client making import deals I need to provide the SP with a url from where they can download the car files. That seems to be missing from the DealRequest struct?

[jcace]

minor point: I think this would be classified an "end to end" deal in our terms because Boost can automatically download it from the provided url. So the user doesn't have to import it.

+1 for this feature though- I think having the flexibility to set transfer params is useful for a lot of use cases 😃

[anjor]

My understanding was end-to-end: client pushes the data when deal is accepted import: SP pulls the data when they want

is that not correct?

[jcace]

if you provide a download URL (http, ftp, etc) in the transfer params, Boost will automatically try to download it as soon as the deal lands. So, the SP doesn't actually decide to pull the data when they want - in that regard it's not an import deal in the traditional sense, SP does not have to do any importing.

I'm thinking of Import deals as those with absolutely no transfer params, so the SP must perform some actions on their end to get the carfile into Boost out of band

At least, this is how it worked when I last tested this 4 months ago- could be different now!

[alvin-reyes]

I started working on this, there is actually a "source" field on the deal request and my intention is to use that as a source on where to get the file. The idea is to add a CID and the source (multiaddr or http or libp2p) then delta pulls it from the source, make an online deal etc.

there's going to be two modes here:

• e2e deal - requires the file to be pulled by delta since it needs to compute for commp, transfer to SP.
• import deal - we require pre-computed commp and just add the transfer params which will pull the data from a different source.

If it's an e2e deal:

• delta needs to pulls the data from the source.
• compute commp
• make a deal.
• The transfer here can be optional since we can set the transfer params to use the "source" field of the deal request and let the miner/sp pull it.

if it's an import deal:

• pre-computed commp is required
• delta just uses the source as transfer param
• make a deal using the transfer param
• The transfer params will use "source" field of the deal request and let the miner/sp pull it.

[anjor]

Would be good to also support http url for source.

This issue is currently the blocker for me to be able to integrate data prep tools with delta.

[alvin-reyes]

https://github.com/application-research/delta/pull/118

[anjor]

Given that delta uses boost under the hood, i don't think this actually supports the out-of-band/import deal flow. Even if this is set on the delta side, boost will never use it.

Which means it doesn't get sent to the SP.

[anjor]

https://github.com/filecoin-project/boost/blob/main/storagemarket/types/types.go#L83

[anjor]

So I think it does do the thing @jcace describes above for e2e deals, but it does not handle import deals.

[alvin-reyes]

the user can use the "label" field to include the source of the CAR file. So the flow would be

1 - create an import deal with a label - "https://myfile.car" 2 - send import deal to miner 3 - miner check the label and download the "https://myfile.car"

Does that make sense?

[anjor]

Hmm maybe, that's a bit hacky though. Also will need each SP to implement their own tooling to parse + handle this.