HTTP Headers Cleanup: API and Gateway

[lidel]

Summary

We should have:

• Interop tests that ensure HTTP responses have the same headers and values, no matter which implementation is the backend

• Diagnostic tool/script that can be run against any HTTP API or Gateway port and provide quick health status

Status

• 2018-Q4: Interop is a mixed bag. Some headers are tested, some are not or different values across implementations. A simple test script exists but is incomplete.

TODO

Go over below headers of interest and ensure proper safeguards are in place.

Gateway

• [ ] X-Ipfs-Path: IPFS Path of returned resource
  • Unicode will probably break some header parsers, user-provided example: ʇɐɔ‾ƃuᴉʞnd.jpg
• [ ] Etag: resolved CID/multihash of returned payload
  • [ ] If-None-Match header
• [ ] Cache-Control:
  • [ ] disable cache for directory listings and errors,
  • [x] enable heavy caching for immutable assets from /ipfs/ namespace
  • [ ] Cache-Control for /ipns/ – https://github.com/ipfs/go-ipfs/issues/1818 / https://github.com/ipfs/go-ipfs/issues/5968 / https://github.com/ipfs/go-ipfs/pull/8074
  • [ ] other cache control fixes https://github.com/ipfs/go-ipfs/issues/8717
  • [ ] Stale-While-Revalidate in browsers https://www.mnot.net/blog/2014/06/01/chrome_and_stale-while-revalidate
• [ ] Suborigin: use root CID in base32 and literal prefix to conform to the current suborigin spec (https://github.com/ipfs/in-web-browsers/issues/66)
• [ ] Last-Modified
  • discuss.ipfs.io: user experiencing last-modified from 1970
  • https://github.com/ipfs/go-ipfs/issues/7968
• [ ] CORS
  • [ ] CORS returning all, instead of just one https://github.com/ipfs/go-ipfs/issues/1659#issuecomment-512241089\
  • [ ] CORS headers missing on shash normalization redirect https://github.com/ipfs/go-ipfs/issues/8501
  • • all issues labeled with topic/CORS in general: https://github.com/ipfs/go-ipfs/labels/topic%2FCORS
• [ ] some gateways (eg. Infura) use custom content sniffing and disable it on the client via X-Content-Type-Options: nosniff which causes CSS files to be returned as text/plain (https://github.com/ipfs-shipyard/ipfs-deploy/issues/86#issue-484313696)
  • this is a strong signal we need to provide better story around content-type, at least for files used on websites (https://github.com/ipfs/unixfs-v2/issues/11)
  • tracking content-type ideas in https://github.com/ipfs/in-web-browsers/issues/152

API

• [ ] Etag & Cache-Control: without this, all content addressed "gets" skip browser cache which results in degraded performance and wasted bandwidth (details: https://github.com/ipfs/go-ipfs/issues/3543)
• [ ] X-Chunked-Output: various API endpoints break without it (eg. https://github.com/ipfs/go-ipfs/issues/5711)
• [ ] CORS
  • [ ] Access-Control-Expose-Header: without this, JS is unable to see X-Chunked-Output in Chrome (https://github.com/ipfs/go-ipfs/issues/5745)
  • [ ] User-Agent in default list of Access-Control-Allow-Headers to follow whatwg/fetch spec (https://github.com/ipfs/go-ipfs/issues/5138)
• [x] Suggestion to set Access-Control-Allow-Credentials should be removed from docs as it is not used and may cause security issues in some setups.
• [ ] CORS preflight broken with custom header https://github.com/ipfs/go-ipfs/issues/7667

Related

• Examples of issues that occur if we don't have regression tests for gateway
  • https://github.com/ipfs-shipyard/ipfs-share-files/issues/17#issuecomment-416766663: Gateway requests responds with 403 - Forbidden
  • https://github.com/ipfs/go-ipfs/issues/5711: Unexpected X-Chunked-Output in /api/v0/version response
• https://github.com/ipfs/js-ipfs/pull/1537: adding X-Ipfs-Path, Etag, Cache-Control, Suborigin to js-ipfs
• https://github.com/ipfs/go-ipfs/issues/3543: Add etag and other caching headers to all content addressed gets in api
• https://github.com/ipfs/go-ipfs/issues/6213: Gateway has unkillable CORS headers
• public-gateway-smoke-test.sh – Small test suite ensuring Public HTTP Gateway behaves correctly, all CORS headers and permissions are in place etc.
• Snapshot of headers in Q4-2018
• -Allow-Headers vs -Expose-Headers

[lanzafame]

/cc @hsanjuan

[lidel]

For the record, in Q4 2018 headers look like this :upside_down_face:

Public Gateway at ipfs.io

$ curl -s -I -X GET https://ipfs.io/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: X-Requested-With, Range, Content-Range, X-Chunked-Output, X-Stream-Output
Access-Control-Expose-Headers: Content-Range, X-Chunked-Output, X-Stream-Output

go-ipfs v0.4.18 defaults

(equal to ipfs config --json API.HTTPHeaders '{}')

$ curl -s -I -X GET http://127.0.0.1:8080/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
Access-Control-Allow-Headers: Content-Range, X-Chunked-Output, X-Stream-Output
Access-Control-Allow-Methods: PUT
Access-Control-Allow-Methods: GET
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Range, X-Chunked-Output, X-Stream-Output

js-ipfs v0.33.1 defaults

(no API.HTTPHeaders in config)

$ curl -s -I -X GET http://127.0.0.1:9090/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR | grep -i Access-Control
access-control-allow-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length
access-control-expose-headers: X-Stream-Output, X-Chunked-Output, X-Content-Length

[lidel]

-Allow-Headers vs -Expose-Headers

There is an important nuance: Access-Control-Allow-Headers tells browsers which headers can be used in XHR CORS request and Access-Control-Expose-Headers whitelists which response headers will be available to be read by JS:

The Access-Control-Allow-Headers response header is used in response to a preflight [HTTP OPTIONS] request which includes the Access-Control-Request-Headers to indicate which HTTP headers can be used during the actual request.

The Access-Control-Expose-Headers response header indicates which headers can be exposed as part of the response by listing their names.