kubo

kubo copied to clipboard

We need to place limits on the bandwidth ipfs uses. We can do this a few different ways (or a combination thereof):

• per peer limiting on each actual connection object
  • pros:
    • low coordination cost (no shared objects between connections)
    • should have lower impact on performance than blindly rate limiting the whole process
  • cons:
    • no flow control between protocols, dht could drown out bitswap traffic
• per subnet limiting
  • pros:
    • avoids rate-limiting LAN/localhost connections.
  • cons:
    • it's not always possible to tell what's "local" (e.g., with IPv6).
• per protocol limiting on each stream
  • pros:
    • should have the lowest impact on system performance of the three options
    • each protocol gets its own slice of the pie and doesnt impact others
  • cons:
    • increased coordination required, need to reference the same limits across multiple streams
    • still makes it difficult to precisely limit the overall bandwidth usage.
• global limiting using a single rate limiter over all connections
  • pros:
    • will successfully limit the amount of bandwidth ipfs uses.
  • cons:
    • ipfs will be quite slow when rate limited in this way

Related Issues:

• https://github.com/ipfs/go-ipfs/issues/2489
• https://github.com/ipfs/go-ipfs/issues/2828
• https://github.com/ipfs/go-ipfs/issues/2917
• https://github.com/ipfs/go-ipfs/issues/2923

Here's two more related issues :) https://github.com/ipfs/go-ipfs/issues/920 https://github.com/ipfs/go-ipfs/issues/1482

This is critical if you want mass adoption. No one is going to risk their own local Internet connection bandwidth unless they can control it. That means using a 3rd party bandwidth limiter in front of IPFS which is just more complexity that isn't necessary.

Perhaps using the alternative C by default with low limits, but when putting IPFS on an "active" state switch to A (or to no limit at all). The "active" state should be when the user is actively downloading, adding or pinning something and some time after that, or when he is using IPFS from some management GUI or JS app.

I was thinking to implement (but never did), a script that alternates, every ~120 seconds between "offline" and "online" mode. It can also read the amount of connections, and restart the client when passes some threshold. Something like:

• Start client "online"
• Wait 120 seconds
• Kill client
• Start client "offline"
• Wait 120 seconds
• Kill client
• [Repeat]

global limiting using a single rate limiter over all connections cons: ipfs will be quite slow when rate limited in this way

Global limiting has my vote. And I'm not sure if this con is true in all cases: bandwidth of course already has a hard limit (the limit of the connection). So if I already have a max of 20mbit down / 2mbit upload, and I limit ipfs to half of this, that is still a decent amount of bandwidth, isn't it?

I think it would be best to do global limitation and then also limit per protocol relative to the global limit. For example let globalLimitUp = 1mbit/sec, globalLimitDown = 2mbit/sec and then every protocol gets its share of the available bandwidth depending on how important it is for ipfs to function properly.

Maybe i misunderstand the problem though, i just came here because i noticed the high use of bandwidth.

700 peers and 3.5 Mbps, both numbers climbing with no end? I am on win10 and [email protected] running the daemon with ipfs daemon --routing=dhtclient.

@guybrush FYI, you can limit the bandwidth usage by turning off the DHT server on your node by passing the --routing=dhtclient flag to your daemon.

This is essential, checking back on this. Without limiting, it's hard for us to package this in projects -> we can't expect end users to accept such a heavy bandwidth requirement.

Please just add an emoji to the issue itself to add your support. Comments in this thread should be reserved for discussion around the implementation of the feature itself.

I've been running an IPFS daemon for years without problems. But with the latest builds in the past couple weeks, I have a lot of delays in trying to load web pages or even ssh into another server. It's now at the point where I have to shut down the IPFS daemon to do some tasks. My stats are below. The bandwidth doesn't look so bad, so why does my network suddenly seem clogged?

$ for p in /ipfs/bitswap/1.1.0 /ipfs/dht /ipfs/bitswap /ipfs/bitswap/1.0.0 /ipfs/kad/1.0.0 ; do echo ipfs stats bw --proto $p && ipfs stats bw --proto $p && echo "---" ; done ipfs stats bw --proto /ipfs/bitswap/1.1.0 Bandwidth TotalIn: 1.1 MB TotalOut: 6.1 kB RateIn: 1.9 kB/s RateOut: 0 B/s

ipfs stats bw --proto /ipfs/dht Bandwidth TotalIn: 41 kB TotalOut: 3.2 kB RateIn: 483 B/s RateOut: 1 B/s

ipfs stats bw --proto /ipfs/bitswap Bandwidth TotalIn: 0 B TotalOut: 0 B RateIn: 0 B/s RateOut: 0 B/s

ipfs stats bw --proto /ipfs/bitswap/1.0.0 Bandwidth TotalIn: 0 B TotalOut: 0 B RateIn: 0 B/s RateOut: 0 B/s

ipfs stats bw --proto /ipfs/kad/1.0.0 Bandwidth TotalIn: 21 MB TotalOut: 1.6 MB RateIn: 164 kB/s RateOut: 8.9 kB/s

@jefft0 thats odd... those stats seem relatively normal. Are you seeing any odd cpu activity? what sort of bandwidth utilization does your OS report from ipfs? Also, how many connections does your node normally have?

Another question is, since you mentioned noticing this on recent builds, does running an older version of ipfs fix the problem?

Also, cc @mgoelzer and @bigs, despite this being on the go-ipfs repo, this is definitely a libp2p issue. Worth getting on the roadmap for sure.

I solved the problem by restarting my Internet router, restarting the computer, wiping the IPFS build directory and rebuilding the current version (but keeping my current ~/.ipfs folder). I know this wasn't very methodical, but I was desperate. Next time I have bandwidths problems I'll try to figure out which one of these causes the problem.

@jefft0 interesting. Thats actually more helpful information than you could have provided, thanks

Also, just so everyone watching this thread is aware, we have implemented a connection manager that limits the total number of connected peers. This can be configured in your ipfs config under Swarm.ConnMgr, see the config docs for more details.

Definitely a fan of the per-protocol limiting. Perhaps this could be handled with a weighting system? Assign weights to protocols and then set global settings (i.e. throttle after this amt of transfer per duration, halt all transfer after this limit within duration.)

Very cool to see progress! How's the bandwidth cap (eg: 50kb/s) coming along? It'd be super useful for our desktop client :)

Are there news about this topic?

Not at the moment. The current recommended approach is to limit bandwidth in the OS.

PSA: if anyone is looking for a good third-party userspace bandwidth shaper, check trickle:

trickle -s -u 50 -d 50 ipfs daemon --routing=dhtclient

I know people have been using it with go-ipfs in the past on Linux for "global limiting using a single rate limiter over all connections"

Trickle has been reported to work on a wide variety of Unix-like operating systems including OpenBSD, NetBSD, FreeBSD, Linux and Sun Solaris, and is by its very nature also architecture agnostic. – Trickle: A Userland Bandwidth Shaper for Unix-like Systems [PDF]

Trickle looks like a good option for Linux. But for Windows, do you have a good option that could be managed by command line too?

@douglasmsi Netlimiter seems to do the trick. Haven't found a command line interface yet

Instead of limiting used bandwidth is there an deep analysis of consumed bandwidth to know if it is an implementation problem, a design problem, a bug etc?

Bumping this, this is pretty contraining when running a node.

This bandwidth usage issue is why I am not running ipfs node anywhere currently, while I used to run it on one laptop and two desktops of which other was almost 24/7.

I think IPFS would be useful for me, but I cannot run it due to how it currently blocks web browsing and affects even mosh badly.

As an update to my previous comment, I have managed to get ipfs running without affecting web browsing or other activities by tweaking the ConnMgr options.

    "ConnMgr": {
      "GracePeriod": "1s",
      "HighWater": 25,
      "LowWater": 5,
      "Type": "basic"
    },

If I understand this correctly, excess connections are given one second before they get cleaned, maximum preferred peer amount is 25 and connections are attempted to get at least to 5 peers. However at the time of writing I have 4 peers and IPFS Companion reports 60 peers if I visit https://ipfs.io/ or otherwise use the daemon.

I am running ipfs daemon with flags --routing=dhtclient --enable-gc and have also removed public IPv4 addresses from swarm as I am always behind CGN or another NAT I don't control and I find IPv4 connections less reliable. I have enabled Quic for curiosity. Interestingly the default ConnMgr options were killing my IPv6 connectivity (https://github.com/ipfs/go-ipfs/issues/3320?), but my router stops sending IPv6 RAs also by itself when left alone for more than a day or it goes powersaving mode, which requires me to reboot it at least once per day.

    "Swarm": [
      "/ip4/127.0.0.1/tcp/4001",
      "/ip4/127.0.0.1/udp/4001/quic",
      "/ip6/::/tcp/4001",
      "/ip6/::/udp/4001/quic"
    ]

I'm really interested in running a node over a cellular connection. Bandwidth is one aspect and I may be oversimplifying here, but if ipfs is based on bittorrent, shouldn't it be possible to specify a maximum amount of traffic too? Won't this accommodate that majority of use cases as far as internet plans are concerned?

IPFS works differently than bittorrent. In absence of central tracker, it relies on as many as possible connected nodes (DHT) to find and retrieve content. This mandates your node to maintain many connections and yes, use expensive cellular traffic. Traffic limiting would only cause your ipfs node to grind to a halt. IMHO, there should be a way to establish leecher/proxy ipfs nodes, to support cellular or other bandwidth limited ipfs users.

IPFS works differently than bittorrent. In absence of central tracker, it relies on as many as possible connected nodes (DHT) to find and retrieve content. This mandates your node to maintain many connections and yes, use expensive cellular traffic.

This is a bit of a simplification. We do need to keep some connections open however, we should be able to:

1. Significantly reduce background traffic.
2. Suspend connections: https://github.com/libp2p/go-libp2p/issues/438.

Etc...

old but relevant discussion: https://github.com/ipfs/go-ipfs/issues/4029