ipfs-webui icon indicating copy to clipboard operation
ipfs-webui copied to clipboard

Import/export CAR DAG archives

Open lidel opened this issue 4 years ago • 7 comments

Given that:

  • ipfs dag import|export exists
  • go-ipfs 0.9+ supports exporting DAG archive via any public gateway (/api/v0/dag/export)
  • tools like ipfs-car and ipfs-content-addressed-archiver make it easier for people to produce/consume content-addressed archives

We should support "DAG import/export" in ipfs-webui somehow.

Import DAG

Import could work the same as "From IPFS" :

2021-06-01--13-01-05

We most likely want to add an explicit "From CAR Archive" item here (personally, I like the idea of a whimsical car icon :)), but "stroke data" from ipfs-css would also do.

Export DAG

Add complimentary "Export DAG" to context menu on Files screen:

2021-06-11--16-08-29

lidel avatar Jun 01 '21 11:06 lidel

Seems like it would be a good idea to do it after https://github.com/ipfs/js-ipfs/issues/2745

Gozala avatar Jun 23 '21 18:06 Gozala

@lidel as I mentioned on https://github.com/ipfs/ipfs-webui/pull/1838#issuecomment-901733616, we'll need to update to the latest HTTP client across WebUI and the Explore page to support the import part. The export is done on #1837. I don't think I should work on the upgrade right now as that would take a long time working only 10h a week. Updating the http client will also require updating everything else connected: multiformats, ipld, multicodecs, etc.

I think we could merge the export button and evaluate how much effort is needed for the import part.

hacdias avatar Sep 01 '21 11:09 hacdias

On ipfs-http-client API update

Agree that upgrading http-client and all the deps (especially ipld-explorer-components) will be a time sink (due to API changes and refactors). I'll block some time on Friday to understand the scope bit better and decide what's next.

On import UX

In regard to CAR Import/export – I am a bit hesitant to introduce export without the ability to import it as well.
Main worry: people will import .car via Import→File and produce garbage blocks (CAR with original DAG wrapped into redundant dag-pb DAG)

I think we not only need Import→DAG Archive but also make Import→File check for magic bytes and do DAG import instead of importing CAR as binary unixfs file.

lidel avatar Sep 02 '21 00:09 lidel

@lidel I agree with your worry. I didn't think about that, but it's a good point.

hacdias avatar Sep 02 '21 13:09 hacdias

Reopening since we also need https://github.com/ipfs/ipfs-webui/pull/1838

lidel avatar Jun 09 '22 22:06 lidel

Add maximum CAR size so output can be split into multiple files. Standard between storage services seems to be accept 100 MB CAR files for uploads.

hsn10 avatar Jul 05 '22 04:07 hsn10

@hsn10 that is an artificial limitation of some specific services. Users of such services can do the splitting with third-party tools like carbites.

We could integrate carbites splitter into ipfs-webui, but it would be pretty expensive to implement, and require additional UI design (how to opt into splitting, how to control splitting parameters, what should be the default). Definitely not in the scope here, but if someone proposes UX design for this in a NEW issue, we can discuss feasibility there :+1:

lidel avatar Jul 06 '22 14:07 lidel