ipfs get doesn't work on files with "]" in the name

[Arlodotexe]

Checklist

• [X] This is a bug report, not a question. Ask questions on discuss.ipfs.io.
• [X] I have searched on the issue tracker for my bug.
• [X] I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

Kubo version: 0.16.0
Repo version: 12
System version: amd64/windows
Golang version: go1.19.1

Config

Click to expand

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Origin": [
        "http://example.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": true,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWGCLGARP16J2Q67FeykYRxoAagJBVKG2c18ggkXQy7zgX"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": [
      {
        "Addrs": [
          "/dnsaddr/node-1.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcFf2FH3CEgTNHeMRGhN7HNHU1EXAxoEk6EFuSyXCsvRE"
      },
      {
        "Addrs": [
          "/dnsaddr/node-2.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcFmLd5ySfk2WZuJ1mfSWLDjdmHZq7rSAua4GoeSQfs1z"
      },
      {
        "Addrs": [
          "/dnsaddr/node-3.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfFmzSDVbwexQ9Au2pt5YEXHK5xajwgaU6PpkbLWerMa"
      },
      {
        "Addrs": [
          "/dnsaddr/node-4.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfJeB3Js1FG7T8YaZATEiaHqNKVdQfybYYkbT1knUswx"
      },
      {
        "Addrs": [
          "/dnsaddr/node-5.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfVvzK4tMdFmpJjEKDUoqRgP4W9FnmJoziYX5GXJJ8eZ"
      },
      {
        "Addrs": [
          "/dnsaddr/node-6.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfZD3VKrUxyP9BbyUnZDpbqDnT7cQ4WjPP8TRLXaoE7G"
      },
      {
        "Addrs": [
          "/dnsaddr/node-7.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfZP2LuW4jxviTeG8fi28qjnZScACb8PEgHAc17ZEri3"
      },
      {
        "Addrs": [
          "/dnsaddr/node-8.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfgsJsMtx6qJb74akCw1M24X1zFwgGo11h1cuhwQjtJP"
      },
      {
        "Addrs": [
          "/dnsaddr/node-9.ingress.cloudflare-ipfs.com"
        ],
        "ID": "Qmcfr2FC7pFzJbTSDfYaSy1J8Uuy8ccGLeLyqJCKJvTHMi"
      },
      {
        "Addrs": [
          "/dnsaddr/node-10.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfR3V5YAtHBzxVACWCzXTt26SyEkxdwhGJ6875A8BuWx"
      },
      {
        "Addrs": [
          "/dnsaddr/node-11.ingress.cloudflare-ipfs.com"
        ],
        "ID": "Qmcfuo1TM9uUiJp6dTbm915Rf1aTqm3a3dnmCdDQLHgvL5"
      },
      {
        "Addrs": [
          "/dnsaddr/node-12.ingress.cloudflare-ipfs.com"
        ],
        "ID": "QmcfV2sg9zaq7UUHVCGuSvT2M2rnLBAPsiE79vVyK3Cuev"
      }
    ]
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "6h",
    "Strategy": "all"
  },
  "Routing": {
    "Methods": null,
    "Routers": null,
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": [],
    "ConnMgr": {
      "GracePeriod": "1m",
      "HighWater": 75,
      "LowWater": 45,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "foo": "foobar"
}

Description

Repro steps:

1. Create an empty folder
2. Create an empty file in that folder, and name it anything. Include ] in the name.
3. Add the folder to Kubo with ipfs add -r ./path
4. Copy the folder CID from the terminal
5. Run ipfs get Qmfoo on the copied CID.
6. Observe the error: Error: invalid platform path: path components cannot contain any of [<>:"\|?*] : "test file].txt"

It's worth noting that this does not happen with the [ character.

[CsterKuroi]

reproduce in old version v0.9.1

[after-egypt]

On the Linux version 0.16.0 I didn't get any errors.

[lidel]

This is because this problem is specific to MS Windows.

Ref. https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file:

The following reserved characters:

    < (less than)
    > (greater than)
    : (colon)
    " (double quote)
    / (forward slash)
    \ (backslash)
    | (vertical bar or pipe)
    ? (question mark)
    * (asterisk)

User-friendly way to resolve this is to

• easy fix: provide user with meaningful error ("Microsoft Windows does not support" instead of vague "invalid platform path".
• proper fix: add ipfs get --with-unsupported-filenames as opt-in (and mention it in the error returned to the user) – passing this parameter will allow Windows users to get data but replace unsupported characters in filenames with underscore (_).

[franklovefrank]

hi @Arlodotexe I am looking at this issue but cannot reproduce it. can you please post a stack trace?

[susarlanikhilesh]

@lidel this is not reproducible on windows.

[Jorropo]

@susarlanikhilesh files do not have names they are just bytes, directories have a list of name cid pairs.

You need to try to download the folder that contains a file with a ] not a file.

[phillmac]

@lidel the problem is, neither the right square bracket or left square bracket are restricted, and you can in fact have files and folder names contain those. the link nor the list of characters you referenced forbid using right and left square bracket.

Can we please get them removed from the filter list?

I specifically want the brackets preserved and not replaced with underscores because they are an important part of folder naming schemas that will break with them removed or replaced.

[lidel]

Ah, indeed. Tracked down the Windows-specific sanitization, it is in go-libipfs/tar (ipfs get RPC uses TAR behind the scenes): https://github.com/ipfs/go-libipfs/blob/302b2799386dea7afb72ba0b4c32a5c427215d06/tar/sanitize_windows.go#L12

Just like others noted the [ and ] are allowed in file names on Windows, but we try to sanitize them for some reason (was unable to find out why, code is 5 years old).

If anyone with Windows has time to test allowing them does not break ipfs get, please submit a fix (open a PR in go-libipfs and here)

[djdv]

Looks like my name is on that commit (oh no, embarrassing ancient code), but that's too long ago for me to remember the full context, and was mostly discussed offline. Also late response because no ping ;^)

I do remember that this was in relation to one of the tons of issues related to get on Windows which was practically unusable in many different ways on Windows at the time. https://blog.ipfs.tech/36-a-look-at-windows/#file-output-names

I imagine if there was any additional characters added on top of Win32's own restrictions, they were probably related to some other portability problems in one of libraries in use at the time and probably isn't necessary to filter out anymore (if it ever was, it's possible those were added by mistake by myself or requested by someone else).

If anyone with Windows has time to test allowing them does not break ipfs get, please submit a fix (open a PR in go-libipfs and here)

This appears to still be a problem. These characters should certainly be allowed if they don't break anything anymore in IPFS or the host OS. I doubt I'll have the ~~motivation~~ availability to tackle this, but if someone else does, you can ping me to get a second person to run tests on a Windows machine. It really should be as simple as removing the characters and running a bunch of checks I guess, some praying and hoping.

[djdv]

Upon inspection, it looks like the original code was using reservedCharsStr as a regex pattern, I.e. [ and ] are not part of the set, they mean "any of" in Regex. The code was changed from: regexp.MustCompile(reservedCharsStr) / reservedCharsRegex.FindAllStringIndex to: strings.ContainsAny by another commit which accidentally includes [ and ] into the filtered set (which they're not supposed to be in).

That explains why it worked originally. The tests for this must have been lost somewhere. I believe originally we were required to do sharness tests in bash or something, which probably got thrown out on Windows? No idea.

[gammazero]

Fixed when updated to use boxo with https://github.com/ipfs/boxo/pull/1047