ipfs
Ipfs kubo node memory usage increases endlessly
Checklist
- [X] This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
- [x] I have searched on the issue tracker for my bug.
- [X] I am running the latest kubo version or have an issue updating.
Installation method
ipfs-desktop
Version
Kubo version: 0.29.0-3f0947b
Repo version: 15
System version: amd64/linux
Golang version: go1.22.4
Config
{
"API": {
"HTTPHeaders": {
"Access-Control-Allow-Methods": [
"PUT",
"POST"
],
"Access-Control-Allow-Origin": [
"http://0.0.0.0:5001"
]
}
},
"Addresses": {
"API": "/ip4/0.0.0.0/tcp/5001",
"Announce": null,
"AppendAnnounce": null,
"Gateway": "/ip4/0.0.0.0/tcp/8080",
"NoAnnounce": [
"/ip4/10.0.0.0/ipcidr/8",
"/ip4/100.64.0.0/ipcidr/10",
"/ip4/169.254.0.0/ipcidr/16",
"/ip4/172.16.0.0/ipcidr/12",
"/ip4/192.0.0.0/ipcidr/24",
"/ip4/192.0.2.0/ipcidr/24",
"/ip4/192.168.0.0/ipcidr/16",
"/ip4/198.18.0.0/ipcidr/15",
"/ip4/198.51.100.0/ipcidr/24",
"/ip4/203.0.113.0/ipcidr/24",
"/ip4/240.0.0.0/ipcidr/4",
"/ip6/100::/ipcidr/64",
"/ip6/2001:2::/ipcidr/48",
"/ip6/2001:db8::/ipcidr/32",
"/ip6/fc00::/ipcidr/7",
"/ip6/fe80::/ipcidr/10"
],
"Swarm": []
},
"AutoNAT": {
"ServiceMode": "disabled"
},
"Bootstrap": [],
"DNS": {
"Resolvers": {}
},
"Datastore": {
"BloomFilterSize": 0,
"GCPeriod": "1h",
"HashOnRead": false,
"Spec": {
"mounts": [
{
"child": {
"path": "blocks",
"shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
"sync": true,
"type": "flatfs"
},
"mountpoint": "/blocks",
"prefix": "flatfs.datastore",
"type": "measure"
},
{
"child": {
"compression": "none",
"path": "datastore",
"type": "levelds"
},
"mountpoint": "/",
"prefix": "leveldb.datastore",
"type": "measure"
}
],
"type": "mount"
},
"StorageGCWatermark": 90,
"StorageMax": "10GB"
},
"Discovery": {
"MDNS": {
"Enabled": false
}
},
"Experimental": {
"FilestoreEnabled": false,
"GraphsyncEnabled": false,
"Libp2pStreamMounting": false,
"OptimisticProvide": false,
"OptimisticProvideJobsPoolSize": 0,
"P2pHttpProxy": false,
"StrategicProviding": false,
"UrlstoreEnabled": false
},
"Gateway": {
"APICommands": [],
"DeserializedResponses": null,
"DisableHTMLErrors": null,
"ExposeRoutingAPI": null,
"HTTPHeaders": {},
"NoDNSLink": false,
"NoFetch": false,
"PathPrefixes": [],
"PublicGateways": null,
"RootRedirect": "",
"Writable": false
},
"Identity": {
"PeerID": "12D3KooWR66ReXcHWSzQEihJpWYUDQgQMAPfnxE3HeGdr1cuLnWS"
},
"Import": {
"CidVersion": null,
"HashFunction": null,
"UnixFSChunker": null,
"UnixFSRawLeaves": null
},
"Internal": {
"Bitswap": {
"EngineBlockstoreWorkerCount": 4096,
"MaxOutstandingBytesPerPeer": null,
"ProviderSearchDelay": null,
}
},
"Ipns": {
"RecordLifetime": "",
"RepublishPeriod": "",
"ResolveCacheSize": 128
},
"Migration": {
"DownloadSources": [],
"Keep": ""
},
"Mounts": {
"FuseAllowOther": false,
"IPFS": "/ipfs",
"IPNS": "/ipns"
},
"Peering": {
"Peers": null
},
"Pinning": {
"RemoteServices": {}
},
"Plugins": {
"Plugins": null
},
"Provider": {
"Strategy": ""
},
"Pubsub": {
"DisableSigning": false,
"Router": ""
},
"Reprovider": {},
"Routing": {
"AcceleratedDHTClient": false,
"Methods": null,
"Routers": null
},
"Swarm": {
"AddrFilters": [
"/ip4/10.0.0.0/ipcidr/8",
"/ip4/100.64.0.0/ipcidr/10",
"/ip4/169.254.0.0/ipcidr/16",
"/ip4/172.16.0.0/ipcidr/12",
"/ip4/192.0.0.0/ipcidr/24",
"/ip4/192.0.2.0/ipcidr/24",
"/ip4/192.168.0.0/ipcidr/16",
"/ip4/198.18.0.0/ipcidr/15",
"/ip4/198.51.100.0/ipcidr/24",
"/ip4/203.0.113.0/ipcidr/24",
"/ip4/240.0.0.0/ipcidr/4",
"/ip6/100::/ipcidr/64",
"/ip6/2001:2::/ipcidr/48",
"/ip6/2001:db8::/ipcidr/32",
"/ip6/fc00::/ipcidr/7",
"/ip6/fe80::/ipcidr/10"
],
"ConnMgr": {},
"DisableBandwidthMetrics": false,
"DisableNatPortMap": true,
"RelayClient": {},
"RelayService": {},
"ResourceMgr": {},
"Transports": {
"Multiplexers": {},
"Network": {},
"Security": {}
}
}
}
Description
Hi, memory usage increases endlessly. The ipfs kubo node is installed on a server with:
- 4 CPU
- 16GB RAM
- 100GB HD
I use docker stats to monitor the memory usage, here are the current log:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
45ddbe02cb58 ipfs 156.94% 4.876GiB / 15.61GiB 31.24% 3.43GB / 1.82GB 26MB / 1.5GB 22
Debugging infos: memory stack: ipfs.stack.tar.gz memory heap: ipfs.heap.tar.gz cpu profile: ipfs.cpuprof.tar.gz
Looks like some goroutines are hanging like:
goroutine 89125206 [select, 99 minutes]:
github.com/quic-go/quic-go.(*incomingStreamsMap[...]).AcceptStream(0x2f23e40, {0x2f04190, 0x3ffe080})
github.com/quic-go/[email protected]/streams_map_incoming.go:82 +0x111
github.com/quic-go/quic-go.(*streamsMap).AcceptUniStream(0xc01c35df10, {0x2f04190, 0x3ffe080})
github.com/quic-go/[email protected]/streams_map.go:192 +0xce
github.com/quic-go/quic-go.(*connection).AcceptUniStream(0x1dd1d07?, {0x2f04190?, 0x3ffe080?})
github.com/quic-go/[email protected]/connection.go:2289 +0x29
github.com/quic-go/quic-go/http3.(*connection).HandleUnidirectionalStreams(0xc026333220, 0xc027924d80)
github.com/quic-go/[email protected]/http3/conn.go:124 +0xa2
created by github.com/quic-go/quic-go/http3.(*SingleDestinationRoundTripper).init in goroutine 89112403
github.com/quic-go/[email protected]/http3/client.go:98 +0x2f2
more infos in the attached memory stack file
I guess your cluster / stack / compose / start command is interesting here as you can limit resources there.
For myself I limited the containers in my swarm because I had no luck doing it with the node's config. I also switched to ipns -> dhtclient to reduce traffic on some nodes. Maybe this also reduces the memory footprint (wild guess).
It seems that, the problem is introduced since 0.28.0, and has been fix in the master branch. So I would revert to 0.27.0 .
@etherscoredao if you're able to try with a custom build try updating to the latest go-libp2p master and seeing if things have improved given the fix here https://github.com/libp2p/go-libp2p/issues/2841
I see this too. Debian Linux - x64
[2657620.399067] Out of memory: Killed process 4187 (ipfs) total-vm:48650700kB, anon-rss:34086188kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:73548kB oom_score_adj:0
