Gateway improvements

[lidel]

This a placeholder for gathering references to create a more fleshed out plan.

We have distinct Gateway (/ipfs, /ipns) and API (/api/v0) endpoints today, and only the most basic read operations can be executed via Gateway.

Exposing /api/v0 on the gateway was a mistake, and we need to work towards removing the need for it.

The goal here would be to shift what is possible on the Gateway endpoints natively, so most popular operations like manipulating IPLD work without touching the advanced API.

---

Ordered improvements

Lower effort, high impact

Things well understood, with clear implementation plan:

• [x] smart HTTP cache invalidation for /ipns/ "dropping only the branches that changed, not entire DAG" (https://github.com/ipfs/go-ipfs/pull/8720)
• [x] block and car stream support (wip: https://github.com/ipfs/go-ipfs/pull/8758)
• [ ] ipld selector support (for car stream download) (https://github.com/ipfs/go-ipfs/issues/8769)
• [ ] dag-json and dag-cbor support (https://github.com/ipfs/in-web-browsers/issues/182)

High impact maintenance work

Things that do not add anything new, but need to happen to solve production issues, improve devexp, and create solid foundations for wider community to participate in Gateway work beyond MVP features listed in the section above.

• [ ] SPECS
• [x] Proper cache-control for /ipns/ based on TTL (https://github.com/ipfs/go-ipfs/issues/1818#issuecomment-1015849462)
• [ ] HTTP headers cleanup (https://github.com/ipfs/in-web-browsers/issues/132)
• [x] extracting gateway code into a standalone library (https://github.com/ipfs/boxo/tree/main/gateway)
• [ ] compliance test suite

Higher effort, high impact

Things that have extremely high impact, but need some additional research / design to happen.

• [ ] writable gateways (https://github.com/ipfs/specs/issues/375)
  • [ ] POST /ipfs/ with support for IPLD content types (as a template for ipfs:// behavior in browsers)
• [ ] mountable gateways, Stage 0: read-only /ipfs/{cid}
  "Mount any gateway CID root as a native read-only drive" (https://github.com/ipfs/in-web-browsers/issues/146)
• [ ] mountable gateways, Stage 1: read-write MFS "Mount Files from ipfs-desktop as a native, writable drive" (https://github.com/ipfs/in-web-browsers/issues/146)

---

Old Brainstorm

Below is the landscape of things worth considering.

No regrets

Things worth doing

• ?format=car|block|... – https://github.com/ipfs/go-ipfs/issues/8234
  • better IPLD support (https://github.com/ipfs/in-web-browsers/issues/182)
  • support IPLD, at very leasst DAG-CBOR/JSON (https://github.com/ipfs/go-ipfs/pull/8037)
  • JSON/CBOR dir listings (https://github.com/ipfs/go-ipfs/issues/7552)
  • ability to fetch a single block
  • ability to fetch arbitrary DAG as CAR
• Revisit the concept of Writable Gateways – https://github.com/ipfs/specs/issues/375 :point_right: what if:
  • not limited to unixfs (#182)
  • supports MFS on localhost
  • supports IPNS publishing via HTTP POST
  • and/or supports DAG imports (#170)
  • can be mounted as network drive (WebDAV: ipfs/kubo#146)
• HTTP headers cleanup (https://github.com/ipfs/in-web-browsers/issues/132)
• Improved image when social sharing a directory, similar to what github does (ipfs/go-ipfs#8811)
• DNSLink/IPNS fallback cache (https://github.com/ipfs/go-ipfs/issues/8245)

Other ideas

Big complexity and low impact for user running it on localhost.

• resumable uploads

  • prompt: importing huge file (DAG, or just a 4K video) over janky connection
  • prior art: https://tus.io/faq.html, https://github.com/ipfs/js-ipfs/pull/1540

• access controls?

  • writable aspect, and also pinning both need some access controls to limit capabilities to specific Origins or requests with correct access token

[RangerMauve]

Regarding being able to POST to a gateway, I think it would be useful to support multipart/form-data in order to upload a bunch of files at once.

E.g. using the FormData API in browsers: https://developer.mozilla.org/en-US/docs/Web/API/FormData

I've done some tests in the Agregore Browser and in the ipfs-protocol-compliance-suite and it's been pretty nice for ergonomics.

Here's what it could look like from a user's perspective:

const formData = new FormData()

formData.append('file', new Blob(['hello world!']), 'example1.txt')
formData.append('file', new Blob(['Goodbye World?']), 'example2.txt')

// Post to empty directory CID
const postResponse = await fetch('ipfs://bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354/', {
  method: 'POST',
  body: formData
})

const newURL = await postResponse.text()

// newURL will be a directory with `example1.txt` and `example2.txt`

This will also play well with people using file inputs or drag and drop to upload data.

[lidel]

Created https://github.com/ipfs/specs/issues/375 for tracking ideas around "Writable Gateways"