service-worker-gateway icon indicating copy to clipboard operation
service-worker-gateway copied to clipboard
verified publisher icon ipfs

Trustless responses: ?format=raw|car|ipns-record

Open agmap opened this issue 1 year ago • 5 comments

https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.<host>/_Metadata.json?format=car --> dweb.link delivers the CAR file only of the filename.json --> inbrowser.link delivers the full CAR file of the full cid directory.

https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.<host>/_Metadata.json?format=raw --> dweb.link downloads file in raw format --> inbrowser.link downloads file in raw format

agmap avatar Sep 12 '24 07:09 agmap

@SgtPooki until we implement this, for now return an explicit 501 Not Implemented error until https://specs.ipfs.tech/http-gateways/trustless-gateway/ is implemented, to avoid user confusion when unsupported ?format=car|raw and Accept: application/vnd.ipld.* is passed in request?

Broken responses make ecosystem feel flaky, clear 501 is better for end users.

Once we get to this, related conformance tests that will help with implementing the spec:

  • https://github.com/ipfs/gateway-conformance/blob/main/tests/trustless_gateway_car_test.go
  • https://github.com/ipfs/gateway-conformance/blob/main/tests/trustless_gateway_raw_test.go

lidel avatar Nov 22 '24 16:11 lidel

FYI this is waiting on https://github.com/ipfs/helia/pull/767 and then we will be good to go

SgtPooki avatar Mar 18 '25 14:03 SgtPooki

So CAR responses are not supported at all currently?

Having trouble with a format=car request on a sub-directory.

You can see what's breaking in this example:

  • Broken: https://dapprank-eth.ipns.inbrowser.link/
  • Working: https://dappdir.eth.limo/

oed avatar Mar 30 '25 20:03 oed

You can see what's breaking in this example:

The broken link you provided doesn't seem relevant to this issue. This issue is discussing the query parameter of ?format=car returning a car file with the appropriate content.

@oed car responses are supported, but there are a few issues:

  1. car file is unnecessarily bloated (too much content, instead of just verification blocks to the necessary content. https://github.com/ipfs/helia/pull/767 fixes this)
  2. query params were not previously preserved (they are now on inbrowser.dev)

SgtPooki avatar Apr 07 '25 19:04 SgtPooki

@SgtPooki ok, because the link above uses ?format=car on a sub directory to retrieve data (as explained here). If it currently returns too much data, or the wrong root I could see how this could cause the issue.

Edit: To be clear, this issue only happens on the service-worker gateway. Not on any Kubo based gateway.

oed avatar Apr 10 '25 11:04 oed

Now ?format=car doesn't create a CAR file with dag-scope=all (including parent directory structure) Inside the CAR file the root directory structure is missing. Test to reproduce https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.inbrowser.dev/_Metadata.json?format=car --> parent directory structure missing https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.dweb.link/_Metadata.json?format=car --> parent directory structure is contained in the CAR according IPIP-0402

agmap avatar May 28 '25 15:05 agmap

@agmap I will get to looking at this today.. I don't think inbrowser.dev nor link have the latest @helia/car fixes (https://github.com/ipfs/helia/pull/767) and new parameters for dag-scope support yet.

SgtPooki avatar May 28 '25 17:05 SgtPooki

FYI work for this is occurring at https://github.com/ipfs/helia-verified-fetch/pull/198, and then i'll get it pulled into service-worker gateway and update inbrowser.dev

SgtPooki avatar May 28 '25 19:05 SgtPooki

OK, there are some other libraries that need to be updated before this will work. I was not aware of that. Thanks for the quick reply and working on that issue.

agmap avatar May 29 '25 06:05 agmap

Fwiw, looks like https://dapprank-eth.ipns.inbrowser.link/ now works properly with the recent changes!

oed avatar May 29 '25 10:05 oed

FYI @agmap car with dagscope support should be working properly at https://inbrowser.dev now that https://github.com/ipfs/service-worker-gateway/actions/runs/15447282773 is done.

We do have an issue where we're including a duplicate root block unnecessarily, but it's no longer the entire dag, so this is a bug, but hopefully only a minor inconvenience. I'm going to close this here and re-open an issue for @helia/car

inbrowser.dev

I downloaded https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.inbrowser.dev/_Metadata.json?format=car and saved as inbrowser_dev-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car

> go-car inspect ~/Downloads/inbrowser_dev-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car

Version: 1
Roots: QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
Root blocks present in data: Yes
Block count: 3
Min / average / max block length (bytes): 246 / 312 / 345
Min / average / max CID length (bytes): 34 / 35 / 36
Block count per codec:
	dag-pb: 3
CID count per multihash:
	sha2-256: 3

> go-car debug ~/Downloads/inbrowser_dev-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car

car compile /Users/sgtpooki/Downloads/inbrowser_dev-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car
root QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
--- bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
+++ dag-json bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
@@ -0,51 +0,51 @@
{
	"Data": {
		"/": {
			"bytes": "CAE"
		}
	},
	"Links": [
		{
			"Hash": {
				"/": "QmP8jTG1m9GSDJLCbeWhVSVgEzCPPwXRdCRuJtQ5Tz9Kc9"
			},
			"Name": "README.txt",
			"Tsize": 1150
		},
		{
			"Hash": {
				"/": "QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v"
			},
			"Name": "_Metadata.json",
			"Tsize": 246
		},
		{
			"Hash": {
				"/": "QmUh6QSTxDKX5qoNU1GoogbhTveQQV9JMeQjfFVchAtd5Q"
			},
			"Name": "albums",
			"Tsize": 61655886776
		},
		{
			"Hash": {
				"/": "QmU7gJi6Bz3jrvbuVfB7zzXStLJrTHf6vWh8ZqkCsTGoRC"
			},
			"Name": "apolloarchivr.py",
			"Tsize": 4024
		},
		{
			"Hash": {
				"/": "QmRSxRRu9AoJ23bxb2pFeoAUFXMAdki7RZu2T7e6zHRdu6"
			},
			"Name": "build_frontend_index.py",
			"Tsize": 9214
		},
		{
			"Hash": {
				"/": "QmeQtZfwuq6aWRarY9P3L9MWhZ6QTonDe9ahWECGBZjyEJ"
			},
			"Name": "frontend",
			"Tsize": 46357163
		}
	]
}

--- bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
+++ dag-json bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
@@ -0,51 +0,51 @@
{
	"Data": {
		"/": {
			"bytes": "CAE"
		}
	},
	"Links": [
		{
			"Hash": {
				"/": "QmP8jTG1m9GSDJLCbeWhVSVgEzCPPwXRdCRuJtQ5Tz9Kc9"
			},
			"Name": "README.txt",
			"Tsize": 1150
		},
		{
			"Hash": {
				"/": "QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v"
			},
			"Name": "_Metadata.json",
			"Tsize": 246
		},
		{
			"Hash": {
				"/": "QmUh6QSTxDKX5qoNU1GoogbhTveQQV9JMeQjfFVchAtd5Q"
			},
			"Name": "albums",
			"Tsize": 61655886776
		},
		{
			"Hash": {
				"/": "QmU7gJi6Bz3jrvbuVfB7zzXStLJrTHf6vWh8ZqkCsTGoRC"
			},
			"Name": "apolloarchivr.py",
			"Tsize": 4024
		},
		{
			"Hash": {
				"/": "QmRSxRRu9AoJ23bxb2pFeoAUFXMAdki7RZu2T7e6zHRdu6"
			},
			"Name": "build_frontend_index.py",
			"Tsize": 9214
		},
		{
			"Hash": {
				"/": "QmeQtZfwuq6aWRarY9P3L9MWhZ6QTonDe9ahWECGBZjyEJ"
			},
			"Name": "frontend",
			"Tsize": 46357163
		}
	]
}

--- QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
+++ dag-json QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
@@ -0,8 +0,8 @@
{
	"Data": {
		"/": {
			"bytes": "CAIS6wF7InRpdGxlIjogIlByb2plY3QgQXBvbGxvIEFyY2hpdmUiLCAibGljZW5zZSI6IHsic291cmNlIjogImh0dHBzOi8vd3d3Lm5hc2EuZ292L211bHRpbWVkaWEvZ3VpZGVsaW5lcy9pbmRleC5odG1sIiwgInN1bW1hcnkiOiAiUHVibGljIERvbWFpbiJ9LCAic291cmNlIjogImh0dHBzOi8vd3d3LmZsaWNrci5jb20vcGhvdG9zL3Byb2plY3RhcG9sbG9hcmNoaXZlLyIsICJsYXN0X3VwZGF0ZSI6IDE0OTMwODMwNjR9GOsB"
		}
	},
	"Links": []
}

dweb.link

I downloaded https://bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy.ipfs.dweb.link/_Metadata.json?format=car and saved as dweb_link-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car

> go-car inspect ~/Downloads/dweb_link-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car

Version: 1
Roots: QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
Root blocks present in data: Yes
Block count: 2
Min / average / max block length (bytes): 246 / 295 / 345
Min / average / max CID length (bytes): 34 / 35 / 36
Block count per codec:
	dag-pb: 2
CID count per multihash:
	sha2-256: 2

> go-car debug ~/Downloads/dweb_link-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car
car compile /Users/sgtpooki/Downloads/dweb_link-bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy__Metadata.json.car
root QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
--- bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
+++ dag-json bafybeiccfclkdtucu6y4yc5cpr6y3yuinr67svmii46v5cfcrkp47ihehy
@@ -0,51 +0,51 @@
{
	"Data": {
		"/": {
			"bytes": "CAE"
		}
	},
	"Links": [
		{
			"Hash": {
				"/": "QmP8jTG1m9GSDJLCbeWhVSVgEzCPPwXRdCRuJtQ5Tz9Kc9"
			},
			"Name": "README.txt",
			"Tsize": 1150
		},
		{
			"Hash": {
				"/": "QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v"
			},
			"Name": "_Metadata.json",
			"Tsize": 246
		},
		{
			"Hash": {
				"/": "QmUh6QSTxDKX5qoNU1GoogbhTveQQV9JMeQjfFVchAtd5Q"
			},
			"Name": "albums",
			"Tsize": 61655886776
		},
		{
			"Hash": {
				"/": "QmU7gJi6Bz3jrvbuVfB7zzXStLJrTHf6vWh8ZqkCsTGoRC"
			},
			"Name": "apolloarchivr.py",
			"Tsize": 4024
		},
		{
			"Hash": {
				"/": "QmRSxRRu9AoJ23bxb2pFeoAUFXMAdki7RZu2T7e6zHRdu6"
			},
			"Name": "build_frontend_index.py",
			"Tsize": 9214
		},
		{
			"Hash": {
				"/": "QmeQtZfwuq6aWRarY9P3L9MWhZ6QTonDe9ahWECGBZjyEJ"
			},
			"Name": "frontend",
			"Tsize": 46357163
		}
	]
}

--- QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
+++ dag-json QmWXShtJXt6Mw3FH7hVCQvR56xPcaEtSj4YFSGjp2QxA4v
@@ -0,8 +0,8 @@
{
	"Data": {
		"/": {
			"bytes": "CAIS6wF7InRpdGxlIjogIlByb2plY3QgQXBvbGxvIEFyY2hpdmUiLCAibGljZW5zZSI6IHsic291cmNlIjogImh0dHBzOi8vd3d3Lm5hc2EuZ292L211bHRpbWVkaWEvZ3VpZGVsaW5lcy9pbmRleC5odG1sIiwgInN1bW1hcnkiOiAiUHVibGljIERvbWFpbiJ9LCAic291cmNlIjogImh0dHBzOi8vd3d3LmZsaWNrci5jb20vcGhvdG9zL3Byb2plY3RhcG9sbG9hcmNoaXZlLyIsICJsYXN0X3VwZGF0ZSI6IDE0OTMwODMwNjR9GOsB"
		}
	},
	"Links": []
}

SgtPooki avatar Jun 04 '25 16:06 SgtPooki

FYI, car duplicate root will be fixed by https://github.com/ipfs/helia-verified-fetch/pull/252

SgtPooki avatar Jun 09 '25 13:06 SgtPooki