Migration to CIDv1 (default base32)

[kyledrake]

Lowercase identifiers are required in order to work with legacy URI, web browser security (sub)origins and the experimental Protocol Handler API. So we are moving to finish support for CIDv1 across the ecosystem, and use it as default (while still supporting CIDv0/Multihash).

There are some complications related to this switch, such as refactoring the blockstore to use multihashes instead of cid identifiers. A number of temporary solutions were proposed that would have made the change simpler with the current design, however it was decided that since this is going to be a significant update, we should focus on a full CIDv1 migration with one identifier change.

This is the parent issue for all issues related to migration to CIDv1 with default base32 (type 'b'). This issue closes when CIDv1b32 is enabled by default for new versions of go-ipfs, js-ipfs, and ipfs-companion.

@kevina @magik6k @lidel @whyrusleeping @diasdavid @alanshaw @olizilla @kyledrake @lgierth @mikeal

Berlin Jul 2018 developer meeting notes: https://github.com/ipfs/developer-meetings/pull/39

Initial proposal: https://github.com/ipfs/go-ipfs/issues/4143

:bird: Bird's-eye view (WIP by @lidel)

Located here: https://gist.github.com/lidel/ceef96f60f30308e3c07be95182c4c88 (most likely out-of-date!)

[Kubuxu]

cc myself

[daviddias]

@kyledrake can you give an update on the status of this migration in the next IPFS All Hands?

[kyledrake]

I will be unavailable for the call unfortunately, I'm moving tomorrow. @lidel do you think you could give a quick update? It doesn't need to be complicated, just a quick summary of progress based on the referenced issues on 337.

[kevina]

@kyledrake @lidel @diasdavid I am the most qualified to give this update for go-ipfs. I will put something in the notes.

[lidel]

js-ipfs v0.33.0 and go-ipfs v0.4.18 shipped with some relevant changes:

This release makes some significant progress towards solving this issue by introducing two features:

(1) [..] ipfs cid base32 command for converting CID to a case intensive encoding required by domain names. This command converts a CID to version 1 and encodes it using base32.

(2) A hack to allow locally looking up blocks associated with a CIDv0 CID using the equivalent CIDv1 CID (or the reverse). This hack will eventually be replaced with a multihash indexed blockstore, which is agnostic to both the CID version and multicodec content type.

– Release Notes for go-ipfs v0.4.18

This PR adds a jsipfs cid command with tools for formatting, converting and discovering properties of CIDs. All the docs are here: https://github.com/ipfs-shipyard/js-cid-tool – https://github.com/ipfs/js-ipfs/pull/1560

The newly added ipfs cid subcommand (1) makes conversion super easy:

# cidv0(b58) → cidv1(b32)
$ ipfs cid base32 QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# cidv1(b58) → cidv1(b32)
$ ipfs cid base32 zb2rhk6GMPQF3hfzwXTaNYFLKomMeC6UXdUt6jZKPpeVirLtV
bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy

[daviddias]

@kyledrake can you compile a status update on this whole endeavour? How far are we from shipping it? Will there be a big launch?

[kyledrake]

@daviddias The referenced issues here show the current state of things pretty well. We're making good progress on better base32 support and are definitely at a point where we want to decide if we're comfortable with planning for a switch.

I would like an opinion from @alanshaw and @kevina on whether they think they're at a point where we would be comfortable migrating to base32, and if there's any blockers or issues on the technical side of things that would make them uncomfortable with switching until resolving. Then we can move to setting dates for transition, getting out announcements, etc.

[daviddias]

Is there a timeline for when to expect this work to be finalized? What's the plan for the launch? How will the users be informed? Any caveats to the migration?

[kyledrake]

@daviddias We're getting closer to the finish line, so now is a good time to start discussing a rough timeline/plan.

1) Release go-ipfs, js-ipfs, ipfs-companion versions with optional cidv1-base32 support

js-ipfs is going to get there with the 0.34 release (https://github.com/ipfs/js-ipfs/issues/1721), and @alanshaw has set 0.36 as the version with default cidv1-base32 as a rough plan (https://github.com/ipfs/js-ipfs/issues/1721#issuecomment-454025894), which I'm OK with. Absolutely incredible work by the js-ipfs team here!

go-ipfs has a pull request pending (https://github.com/ipfs/go-ipfs/pull/5789) that will bring go-ipfs up to speed. I'm really excited to see this merged and finally put my hshca hack to rest. @kevina has done a ton of work on this (and had to re-do a lot of that work for various reasons) so kudos to him and everyone else at go-ipfs that has put a lot of good work into this.

I'm not sure what the timeline looks like for getting this in go-ipfs yet, @kevina @Stebalien is this something you foresee getting merged into master soon?

I'll let @lidel give us an update on ipfs-companion, it will be better than mine. Tracker issue for ipfs-companion is https://github.com/ipfs-shipyard/ipfs-companion/issues/527

Dates: No hard dates for these releases, but I would like to see them released within the next month if it's not too inconvenient. Most of the work is done here.

Step 3 happens after we get releases of js-ipfs, go-ipfs, ipfs-companion which have the (optional, with arguments) support. Step 2 can be done in parallel:

2) Prepare the gateway for subdomains (*.dweb.link)

This will also be a good point to encourage users to start using the subdomains instead of the ipfs.io gateway, *.ipfs.dweb.link and *.ipns.dweb.link (cc @eefahy @mburns), getting content/apps on their own security origin. But we'll need to make sure whatever we choose works well with ipfs-companion (we good here @lidel?)

The main missing pieces right now:

• The *.ipfs.dweb.link and *.ipns.dweb.link need SSL certificates (https://test.ipfs.dweb.link returns a bad cert)
• A default redirect to HTTPS from HTTP
• An HSTS preload header so we can get them added to the browser lists.

Dates: would be nice to see this done within the next month as well.

3) Blog post with new cidv1-base32 information

We will write a blog post indicating that we will be switching to cidv1-base32, why we are making this change, and how users can start using it now with optional arguments. This will get people more comfortable working with it, show them the new gateway URLs they can (really should!) use, and give them time to prepare.

We absolutely need to release both go-ipfs and js-ipfs (also ipfs-companion?) at the same time with the new default support, so a set date will eventually be needed.

Dates: I will publish this very soon after the optional support releases are made. 2-4 days to complete.

4) Update documentation to reflect use of cidv1-base32

When we release with the new defaults, we are going to have a lot of documentation across the organization that references the old output. We will need to prepare the documentation to reflect these changes when we do the release, particularly the docs portal.

Dates: Work will begin soon after optional support releases are made. Rough, hardly educated guess is 2 weeks of work.

5) Release go-ipfs, js-ipfs, ipfs-companion, documentation versions with cidv1-base32 support

We release everything at the same time, and add to a blog post announcing switch to cidv1-base32 and re-hashing a lot of what was stated in the first blog post.

Dates: I'm not sure about a hard date for final release, it really depends on how comfortable we are with the current progress, I don't want to rush anybody. To throw out a few random dates: Spring Equinox is on March 20th and Summer Solstice is June 21st. I'm leaning towards a date closer to Solstice, as it would give us more room to fix any lingering issues and get some user testing in. But if we're feeling comfortable with progress, we can change to an earlier date. I think we'll have a better feel for things once Step 1 is finished.

---

Let's call this "proposed rough plan". Feel free to add any suggestions/missing info.

[kyledrake]

Adding a missing piece from my writeup:

go-ipfs 0.4.18 shipped with some ipfs cid changes for cidv1-base32. It gives us some clever ways to use it with the currently released version using pipes / additional API calls. For example:

$ echo "hello world" | ipfs add -q | ipfs cid base32
bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
$ ipfs cat bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
hello world
$ curl http://bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby.ipfs.dweb.link
hello world

So if used this way, you could technically start using cidv1-base32 with go-ipfs already (for a lot of common uses).

More info from the blog post: https://blog.ipfs.io/53-go-ipfs-0-4-18/#cidv1-base32-migration

[lidel]

ipfs-companion detects <cidv1b32>.ipfs.foo.tld and displays page actions:

When js-ipfs/go-ipfs backends switch their defaults to cidv1b32, ipfs-companion will support it automatically.

Ad. preparing gateway for subdomains, there is an open question about how ipfs-companion should handle redirects of CID-in-Subdomain and DNSLink websites: https://github.com/ipfs-shipyard/ipfs-companion/issues/667 ← would appreciate feedback on proposed solution.

[kyledrake]

Heads up that @lidel is now the coordinator for this work.

[lidel]

:trophy: dweb.link is available over HTTPS!

Demo: https://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.dweb.link

(thanks @scout!)

:information_source: Plan to support CID-in-subdomain via HTTP Proxy

http://<cidv1b32>.ipfs.localhost

To provide Origin-based isolation for websites loaded from local gateway we are planning to use a special domain names in .localhost space. In the long term, we could move to a dedicated Special-Use Domain Name .ipfs.dweb.

We don't want to mess with people's DNS servers, so the cross-platform support for this relies on the idea of on-the-fly domain support via go-ipfs as HTTP proxy (proposed in https://github.com/ipfs/go-ipfs/issues/5982, https://github.com/ipfs/js-ipfs/issues/1877) + ipfs-companion setting go-ipfs a proxy for specific requests.

If proxied hostname is customizable, the same HTTP Proxy mode could be used on public gateways such as dweb.link: https://github.com/ipfs/infra/issues/81#issuecomment-467552053

:information_source: WIP in js-ipfs: change defaults for adding content to cidv1b32

List of PRs in https://github.com/ipfs/js-ipfs/issues/1440#issuecomment-461049347

:information_source: WIP in go-ipfs: switching DHT (& datastores) to raw multihashes

Some relevant go-ipfs updates were discussed at Golang Core Dev Team Weekly Sync 2019-02-11. See Base32/CIDv1 section or TL;DR:

• Q: What's the priority on this Base32/CIDv1 list? A: Most important next milestone is storing/fetching [raw] multihashes: (Issue on DHT, WIP datastore changes)

[lidel]

PSA: Switching CIDv1 to Base32

We are in the process of switching text representation of CIDv1 from Base58btc to Base32. Details in https://github.com/ipfs/go-ipfs/issues/6220 and https://github.com/ipfs/js-ipfs/issues/1995

Note: This is only about changing the default text representation of CIDv1. CIDv0 in Base58btc is still the default. CIDv1 remains an opt-in.

[lidel]

PSA: Adding support for CIDv1 in IPNS

We are adding a new multicodec to unlock PeerID/KeyID-in-CIDv1 in subdomains:

http://<keyid-in-cidvb32>.ipns.localhost

More details in https://github.com/multiformats/multicodec/issues/130 & https://github.com/ipfs/go-ipfs/issues/5287

[lidel]

Cool update: Cloudflare officially supports CIDv1B32 in subdomains: https://blog.cloudflare.com/continuing-to-improve-our-ipfs-gateway/#subdomain-based-gateway

Example: https://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq.cf-ipfs.com/wiki/

[lidel]

PSA: Opened RFC (change to libp2p SPEC) to switch default text representation of PeerIDs to CID and make Base32 the default mutlibase: https://github.com/libp2p/specs/pull/209 This upstream change is part of effort to unlock use of libp2p-keys in subdomains (https://github.com/ipfs/go-ipfs/issues/5287).

[hsanjuan]

I am transferring this issue to ipfs/specs as we want to clean up this repo.

Progress in Go (the missing part) can be followed here: https://github.com/ipfs/go-ipfs/issues/4143

[lidel]

:mega: PSA: go-ipfs 0.5.0 will bring native support for PeerIDs as CIDv1 & CID-in-subdomain

TL;DR Support for PeerIDs as CIDv1 (https://github.com/ipfs/go-ipfs/issues/5287) and native support for CID-in-subdomain (AKA subdomain gateways) landed in go-ipfs (https://github.com/ipfs/go-ipfs/pull/6096) and will ship with go-ipfs 0.5.0 (https://github.com/ipfs/go-ipfs/issues/7109)

HTTP Gateway

When subdomain gateway is used as a path one, go-ipfs will automatically convert case-sensitive identifiers to base32 and redirect to a subdomain URL. For example, given a subdomain gateway enabled on localhost:8080:

http://localhost:8080/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn → HTTP 301 → http://bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354.ipfs.localhost:8080

When /ipns/ path is used, gateway will convert to CIDv1 with libp2p-key multicodec.

Instructions how to run a subdomain gateway can be found in go-ipfs/docs/config.md:

• Gateway.PublicGateways: UseSubdomains
• Gateway recipes

CLI / RPC API

Command line and RPC API provide handy error message when an invalid multicodec is used in CIDv1:

# resolve Base58 PeerID
$ ipfs name resolve -r /ipns/QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# convert PeerID to CIDv1 (with invalid multicoded, to illustrate next step)
$ ipfs cid base32 QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with dag-pb multicodec
$ ipfs name resolve -r /ipns/bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
Error: peer ID represented as CIDv1 require libp2p-key multicodec: retry with /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with libp2p-key multicodec suggested in previous step
$ ipfs name resolve -r /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

[lidel]

:spider_web: :spider: :tophat: brushing off the dust

• DONE: CIDv0 creation was removed from go-ipfs 0.10 DAG API
  • tldr: ipfs dag put produces CIDv1 with dag-cbor codec by default, but one can tweak the put behavior via --input-codec and --store-codec options.
• IN PROGRESS: Switch to raw multihashes in datastores is planned for go-ipfs 0.12, after we are done with that we can start switching the defaults in the rest of IPFS ecosystem

[lidel]

go-ipfs 0.12 shipped the blockstore migration from full CID to Multihash keys.

Finally, we can do the thing! :scream_cat:

Started poking at go-ipfs to produce CIDv1 by default – tracking remaining work in https://github.com/ipfs/go-ipfs/issues/4143#issuecomment-1092326127