js-ipfs-http-client wrongly reads empty directories as a file

[RangerMauve]

• Version:

This is using v56.0.1 of js-ipfs-http-client

• Platform:

Linux meerkat 5.15.15-76051515-generic #202201160435~1642693824~21.10~97db1bb SMP Thu Jan 20 17:35:05 U x86_64 x86_64 x86_64 GNU/Linux

• Subsystem:

js-ipfs-http-client?

Severity:

Medium

Description:

I was using empty directory CIDs as part of a flow for creating folders with MFS via a go-ipfs daemon.

Whenever I'd try to do an ipfs.ls call on the empty folder CID, it would wrongly read it as being an empty file.

I expected it to yield an empty list.

Steps to reproduce the error:

I've created a test repo here: https://github.com/RangerMauve/js-ipfs-empty-directory-test

• Set up [email protected]
• Run a node via ipfsd-ctl
• Invoke ipfs.ls('/ipfs/bafyaabakaieac')
• It will yield a list with one item that is type: file even though the CID points to an empty folder.

[welcome[bot]]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[lidel]

@RangerMauve why are you using bafyaabakaieac instead of bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku ?

bafyaabakaieac has some data inside, which is why it igets interpreted as a file:

$ ipfs block get bafyaabakaieac | xxd
00000000: 0a02 0801
$ ipfs dag get --output-codec=dag-json bafyaabakaieac | jq
{
  "Data": {
    "/": {
      "bytes": "CAE"
    }
  },
  "Links": []
}

Real empty dir had no data:

$ ipfs block get bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku | xxd
$ ipfs dag get --output-codec=dag-json bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku | jq
{
  "Data": null,
  "Links": []
}

[RangerMauve]

@RangerMauve why are you using bafyaabakaieac instead of bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku ?

bafyaabakaieac has some data inside, which is why it igets interpreted as a file:

$ ipfs block get bafyaabakaieac | xxd
00000000: 0a02 0801
$ ipfs dag get --output-codec=dag-json bafyaabakaieac | jq
{
  "Data": {
    "/": {
      "bytes": "CAE"
    }
  },
  "Links": []
}

Real empty dir had no data:

$ ipfs block get bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku | xxd
$ ipfs dag get --output-codec=dag-json bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku | jq
{
  "Data": null,
  "Links": []
}

Honestly. I'm not sure how bafyaabakaieac got generated. I was asking around on Discord and somebody sent it to me as an example of an empty directory that's inline.

However, when I inspect it in the "Explore" tab inside IPFS Desktop I get the following:

{type: "directory", data: undefined, blockSizes: Array[0]}

which appears to be a directory without additional data.

I think this is due to bytes: "CAE" being the minimal representation for an empty directory in the IPFS filesystem. (can't find the spec).

This seems to be using the "identity" hash function. https://github.com/multiformats/multihash/issues/13

Is there a difference between what the IPFS DAG API does and what the IPFS directory resolving code does?

[rvagg]

yeah .. so the DAG API is giving you the raw data model directly out of the dag-pb codec, which is before UnixFS interpreation. CAE is the base64 bytes of a protobuf structure that says "this is a directory" when passed through UnixFS: https://github.com/ipfs/specs/blob/master/UNIXFS.md#data-format (0x0801 which is protobuf for "field 1, value 1"), which is what you're seeing in the IPFS Desktop.

But:

It will yield a list with one item that is type: file even though the CID points to an empty folder.

is kind of odd, this is a directory, but it doesn't have any links, so shouldn't have any files, so maybe an actual bug here with ls? This would probably be worth someone's time diving down into ls to see what it's getting that makes it report that there's a file in a directory there. There's a possibility that the recent work we did on the DAG API, or older work replacing the dag-pb codec, has impacted other parts of the system that we didn't notice.

[rvagg]

go-ipfs seems to see the expected thing:

$ ipfs ls -v bafyaabakaieac
Hash Size Name
$ ipfs get bafyaabakaieac
Saving file(s) to bafyaabakaieac
 4 B / 4 B [======================================================================================================================================] 100.00% 0s
$ ls -al bafyaabakaieac/
total 8
drwxr-xr-x 2 rvagg rvagg 4096 Apr 20 15:11 .
drwxr-xr-x 9 rvagg rvagg 4096 Apr 20 15:11 ..

[whizzzkid]

js-ipfs is being deprecated in favor of Helia. You can follow the migration plan here https://github.com/ipfs/js-ipfs/issues/4336 and read the migration guide.

Please feel to reopen with any comments before 2023-06-05. We will do a final pass on reopened issues afterward (see https://github.com/ipfs/js-ipfs/issues/4336).