kubo bug: ipfs daemon constantly returns `Error: resource temporarily unavailable` after failing to fetch repo migration files from gateway

Checklist

[X] This is a bug report, not a question. Ask questions on discuss.ipfs.io.
[X] I have searched on the issue tracker for my bug.
[X] I am running the latest kubo version or have an issue updating.

Installation method

third-party binary

Version

╰─ ✘ 1 ❯ ipfs version --all
Kubo version: 0.18.0-675037721
Repo version: 13
System version: arm64/darwin
Golang version: go1.19.5

Config

╰─ ✔ ❯ ipfs config show
{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5002",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8081",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport",
      "/ip4/0.0.0.0/udp/4002/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
  ],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWMBeKmccSU6Jx69sZgCNBKdWwkvM8u3mBN8XEyzG4jLWY"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {
      "Estuary": {
        "API": {
          "Endpoint": "https://api.estuary.tech/pinning"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      },
      "Pinata": {
        "API": {
          "Endpoint": "https://api.pinata.cloud/psa"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      },
      "Web3_Storage": {
        "API": {
          "Endpoint": "https://api.web3.storage"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      }
    }
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {
      "GracePeriod": "1m",
      "HighWater": 40,
      "LowWater": 20,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {
        "WebTransport": true
      },
      "Security": {}
    }
  }
}

Description

I attempted to run ipfs daemon locally, which triggered a migration, and ipfs.io gateway was returning 429's (see https://github.com/protocol/bifrost-infra/issues/2310), which has caused ipfs daemon to fail to start now.

using `ipfs` command from brew, installed yesterday

╰─ ✔ ❯ where ipfs
/opt/homebrew/bin/ipfs

╰─ ✘ 1 ❯ brew info ipfs
Warning: Treating ipfs as a formula.
==> ipfs: stable 0.18.0 (bottled)
Peer-to-peer hypermedia protocol
https://ipfs.tech/
/opt/homebrew/Cellar/ipfs/0.18.0 (11 files, 65.5MB) *
  Poured from bottle on 2023-01-23 at 10:16:26 # <--- installed yesterday at about 10am PST
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/ipfs.rb
License: MIT and (MIT or Apache-2.0)
==> Dependencies
Build: go ✔
==> Caveats
To start ipfs now and restart at login:
  brew services start ipfs
==> Analytics
install: 793 (30 days), 2,957 (90 days), 14,625 (365 days)
install-on-request: 788 (30 days), 2,940 (90 days), 14,544 (365 days)
build-error: 2 (30 days)

Initial `ipfs daemon` run with migration logs

╰─ ✔ ❯ ipfs daemon
Initializing daemon...
Kubo version: 0.18.0-675037721
Repo version: 13
System version: arm64/darwin
Golang version: go1.19.5
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
Looking for suitable migration binaries.
Need 1 migrations, downloading.
Downloading migration: fs-repo-12-to-13...
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/versions"
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/versions"
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/versions"
Error fetching: exceeded number of retries. last error was GET https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/versions error: 429 Too Many Requests: <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html>


Computing default go-libp2p Resource Manager limits based on:
    - 'Swarm.ResourceMgr.MaxMemory': "17 GB"
    - 'Swarm.ResourceMgr.MaxFileDescriptors': 122880

Applying any user-supplied overrides on top.
Run 'ipfs swarm limit all' to see the resulting limits.

Fetching with IPFS: "fs-repo-12-to-13/versions"
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/v1.0.0/fs-repo-12-to-13_v1.0.0_darwin-arm64.tar.gz"
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/v1.0.0/fs-repo-12-to-13_v1.0.0_darwin-arm64.tar.gz"
Fetching with HTTP: "https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/v1.0.0/fs-repo-12-to-13_v1.0.0_darwin-arm64.tar.gz"
Error fetching: exceeded number of retries. last error was GET https://ipfs.io/ipfs/Qmf4yftD4LuMo8JMNPqqw3BtUwYd2VkXMiAThuPE6usrbQ/fs-repo-12-to-13/v1.0.0/fs-repo-12-to-13_v1.0.0_darwin-arm64.tar.gz error: 429 Too Many Requests: <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>openresty</center>
</body>
</html>

Fetching with IPFS: "fs-repo-12-to-13/v1.0.0/fs-repo-12-to-13_v1.0.0_darwin-arm64.tar.gz"
Downloaded and unpacked migration: /var/folders/bl/_gl5_59s11v7qz5ysd6bfgb00000gn/T/migrations1786708364/fs-repo-12-to-13 (v1.0.0)
Running migration fs-repo-12-to-13 ...
  => Running: /var/folders/bl/_gl5_59s11v7qz5ysd6bfgb00000gn/T/migrations1786708364/fs-repo-12-to-13 -path=/Users/sgtpooki/.ipfs -verbose=true
applying 12-to-13 repo migration
locking repo at "/Users/sgtpooki/.ipfs"
  - verifying version is '12'
> Upgrading config to new format
Swarm.ConnMgr settings are different than the old defaults, skipping
updated version file
Migration 12 to 13 succeeded
Success: fs-repo migrated to version 13.

Error: resource temporarily unavailable

`ipfs daemon` attempts after this:

╰─ ✘ 1 ❯ ipfs daemon
Initializing daemon...
Kubo version: 0.18.0-675037721
Repo version: 13
System version: arm64/darwin
Golang version: go1.19.5

Error: resource temporarily unavailable

Jan 24 '23 22:01 SgtPooki

Note that I attempted to run using the latest go version built from master (make build) and am experiencing the same:

╭─    ~/code/work/protocol.ai/ipfs/kubo    master ?2
╰─ ✘ INT ❯ git log --oneline | head -n1
9327ee64c fix: clarity: no user supplied rcmgr limits of 0 (#9563)

╭─    ~/code/work/protocol.ai/ipfs/kubo    master ?2
╰─ ✔ PIPE|0 ❯ ./cmd/ipfs/ipfs daemon
Initializing daemon...
Kubo version: 0.19.0-dev-9327ee64c
Repo version: 13
System version: arm64/darwin
Golang version: go1.18.4

Error: resource temporarily unavailable

Jan 24 '23 22:01 SgtPooki

i run ipfs on a kubernetes, yaml as follow:

---
kind: ReplicaSet
metadata:
  annotations:
    deployment.kubernetes.io/desired-replicas: '4'
    deployment.kubernetes.io/max-replicas: '5'
  labels:
    k8s.kuboard.cn/name: ipfs-deployment
    pod-template-hash: 57d846dddb
  name: ipfs-deployment-57d846dddb
  namespace: default
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: Deployment
      name: ipfs-deployment
      uid: ea17db4d-84f9-4a0b-911e-26a427ac8f8e
  resourceVersion: '111056'
spec:
  replicas: 0
  selector:
    matchLabels:
      k8s.kuboard.cn/name: ipfs-deployment
      pod-template-hash: 57d846dddb
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s.kuboard.cn/name: ipfs-deployment
        pod-template-hash: 57d846dddb
    spec:
      containers:
        - env:
            - name: IPFS_LOGGING
              value: INFO
          image: ipfs/go-ipfs
          imagePullPolicy: IfNotPresent
          name: ipfs
          ports:
            - containerPort: 4001
              protocol: TCP
            - containerPort: 5001
              protocol: TCP
            - containerPort: 8080
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /data/ipfs
              name: volume-y2p8h
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: volume-y2p8h
          nfs:
            path: /home/kube/data
            server: 192.168.138.46
status:
  observedGeneration: 10
  replicas: 0

occured a problem:

Changing user to ipfs ipfs version 0.22.0 Found IPFS fs-repo at /data/ipfs Initializing daemon... Kubo version: 0.22.0-3f884d3 Repo version: 14 System version: amd64/linux Golang version: go1.19.10 Error: resource temporarily unavailable

Sep 27 '23 05:09 xueshuai0922

kubo kubo copied to clipboard

bug: ipfs daemon constantly returns `Error: resource temporarily unavailable` after failing to fetch repo migration files from gateway

Checklist

Installation method

Version

Config

Description

using ipfs command from brew, installed yesterday

Initial ipfs daemon run with migration logs

ipfs daemon attempts after this:

kubo
kubo copied to clipboard

using `ipfs` command from brew, installed yesterday

Initial `ipfs daemon` run with migration logs

`ipfs daemon` attempts after this: