Limit number of outgoing connections

[To-Azamat]

Is your feature request related to a problem? Please describe

When active and crawling BM opened thousands of outgoing UDP connections triggering ISP limit and making internet access for other clients unreachable.

Describe the solution you'd like

Add a setting to limit a number of active outgoing connections.

[denniskrol]

https://github.com/bitmagnet-io/bitmagnet/issues/299

[felipemarinho97]

This may be a good solution

[demogorgonz]

Any other options instead of reducing dht_crawler.scaling_factor ?

I tried setting it to 5 and after some time got same results as https://github.com/bitmagnet-io/bitmagnet/issues/299#issuecomment-2251794323 :

the number of connections went up again in about 2 hours

Just tried setting the value to 1 and will see how it goes.

Saw that VPN might be possible fix but that doesn't seem like great solution..

[demogorgonz]

Ok this seems to work in docker compose:

services:
  bitmagnet:
    image: ghcr.io/bitmagnet-io/bitmagnet:latest
    ulimits:
      nofile:
        soft: 1024
        hard: 2048
    container_name: bitmagnet
    ports:
      # API and WebUI port:
      - "3333:3333"
      # BitTorrent ports:
      - "3334:3334/tcp"
      - "3334:3334/udp"
    restart: unless-stopped
    environment:
      - POSTGRES_HOST=postgres
      - POSTGRES_PASSWORD=XXXXXXXXXXX
      - DHT_CRAWLER_SCALING_FACTOR=2
...

By setting low ulimits my router is not bombarded with UDP connections, this part exactly:

    ulimits:
      nofile:
        soft: 1024
        hard: 2048

Also reduced DHT_CRAWLER_SCALING_FACTOR to 2, will monitor how it behave.

[To-Azamat]

ulimits:
  nofile:
    soft: 1024
    hard: 2048

Are those documented somewhere?

[demogorgonz]

ulimits:
  nofile:
    soft: 1024
    hard: 2048

Are those documented somewhere?

Documented where? BitMagnet, Docker, or Linux ? For Docker and Linux use google/search for general info. Feel free to play with values too, this is just a way to limit number of connections in container until this get fixed in BitMagnet/DHT_Crawler.

[DerBunteBall]

See

man ulimit

on your system for technical documentation about the Linux basics.

Compose Reference for docker-compose specific stuff. It's a service level element. Just search for ulimits here.

[To-Azamat]

ulimits:
  nofile:
    soft: 1024
    hard: 2048

It works, partly. I set those to 512-1024 and also a rule at router firewall to drop any outgoing UDP connections from bitmagnet exceeding 1024. Got 170k dropped connections in 9 hours. Still better than before, but hardly a solution.

[demogorgonz]

ulimits:
  nofile:
    soft: 1024
    hard: 2048

It works, partly. I set those to 512-1024 and also a rule at router firewall to drop any outgoing UDP connections from bitmagnet exceeding 1024. Got 170k dropped connections in 9 hours. Still better than before, but hardly a solution.

I have ended up with following values:

    ulimits:
      nofile:
        soft: 512
        hard: 768

My network seems stable, no more disconnects and other issues.

Edit: Agreed it is not a proper solution, we need config/parameter to limit number of connections in BitMagnet.

[samwathegreat]

This is a real problem for me. Bitmagnet is maxing my openwrt instance out with connections. I've upped the conntrack but it's still not enough and is causing problems for my other devices. 15,000+ simultaneous connections is too much!

We can limit number of max connections with bittorrent, so why not bitmagnet? Should be simple to implement, right?

I'm not using docker (using LXC) so the ulimits option doesn't work for me...

Any advice?

[mgdigital]

Is anyone having these issues with OpenWRT ALSO running through Gluetun VPN, and if not does that help at all? I'll look at making max connections configurable but I suspect this might help.

[demogorgonz]

This is a real problem for me. Bitmagnet is maxing my openwrt instance out with connections. I've upped the conntrack but it's still not enough and is causing problems for my other devices. 15,000+ simultaneous connections is too much!

We can limit number of max connections with bittorrent, so why not bitmagnet? Should be simple to implement, right?

I'm not using docker (using LXC) so the ulimits option doesn't work for me...

Any advice?

You have 2 options:

1. Set limits in /etc/lxc/<container-name>.conf - lxc.prlimit.nofile = 512 768

---

2. Systemd within LXC container :

[Service]
LimitNOFILE=768:512

Do not forget to :

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart your-service

[samwathegreat]

This is a real problem for me. Bitmagnet is maxing my openwrt instance out with connections. I've upped the conntrack but it's still not enough and is causing problems for my other devices. 15,000+ simultaneous connections is too much! We can limit number of max connections with bittorrent, so why not bitmagnet? Should be simple to implement, right? I'm not using docker (using LXC) so the ulimits option doesn't work for me... Any advice?

You have 2 options:

1. Set limits in /etc/lxc/<container-name>.conf - lxc.prlimit.nofile = 512 768

2. Systemd within LXC container :

[Service]
LimitNOFILE=768:512

Do not forget to :

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart your-service

This works for my LXC container with a small modification. The container wouldn't start until I changed the syntax in the .conf file to: lxc.prlimit.nofile = 512:768 (apparently the colon is necessary)

Out of curiosity, is the consensus that these values are optimal? Considering that it was using 15K+ connections before the change, will crawling speed be drastically reduced, or were the excessive connections mostly wasted resources?

Do we expect a connections limit setting to be included in a future version of bitmagnet, or is the solution considered the correct way of dealing with the issue?

Thanks a bunch for the help

[demogorgonz]

This works for my LXC container with a small modification. The container wouldn't start until I changed the syntax in the .conf file to: lxc.prlimit.nofile = 512:768 (apparently the colon is necessary)

That was typo on my end, nice catch! I am glad that you worked it out.

Out of curiosity, is the consensus that these values are optimal? Considering that it was using 15K+ connections before the change, will crawling speed be drastically reduced, or were the excessive connections mostly wasted resources?

Per my testing on my router (ZTE F680), values 512:768 seems stable, but feel free to experiment with higher values if your router can handle it. To be honest i haven't measure crawling speed or impact of reduced connections, but it seems that BitMagnet is working ok, i am sitting at 10MIL+ torrents.

Do we expect a connections limit setting to be included in a future version of bitmagnet, or is the solution considered the correct way of dealing with the issue?

I really hope so the maintainer will include it as config with some lower default values. This is "hacky" way just to restrict number of connections and not so elegant per my personal opinion.

Thanks a bunch for the help ❤ 👍

[samwathegreat]

This works for my LXC container with a small modification. The container wouldn't start until I changed the syntax in the .conf file to: lxc.prlimit.nofile = 512:768 (apparently the colon is necessary)

That was typo on my end, nice catch! I am glad that you worked it out.

Out of curiosity, is the consensus that these values are optimal? Considering that it was using 15K+ connections before the change, will crawling speed be drastically reduced, or were the excessive connections mostly wasted resources?

Per my testing on my router (ZTE F680), values 512:768 seems stable, but feel free to experiment with higher values if your router can handle it. To be honest i haven't measure crawling speed or impact of reduced connections, but it seems that BitMagnet is working ok, i am sitting at 10MIL+ torrents.

Do we expect a connections limit setting to be included in a future version of bitmagnet, or is the solution considered the correct way of dealing with the issue?

I really hope so the maintainer will include it as config with some lower default values. This is "hacky" way just to restrict number of connections and not so elegant per my personal opinion.

Thanks a bunch for the help ❤ 👍

EDIT:

Actually, this doesn't seem to be helping. While I am indeed able to set the prlimit (and it's reflected in the container), the open connections is still totally out of control (currently around ~20k, even with these settings)

See here:

Even though it reflects "nofile" set to 1024:2048 which I'm currently trying, the open connections are insane! I rebooted the openwrt VM and the bitmagnet container to start fresh at zero connections. It's grown to 20k in just a few minutes...

Not sure how I can manage to control this...

I wonder why it reflects the value but isn't honoring it. It is maybe because these limits are 'per process'? Is bitmagnet running multiple simultaneous processes?

#confused

[mgdigital]

Do we expect a connections limit setting to be included in a future version of bitmagnet, or is the solution considered the correct way of dealing with the issue?

I really hope so the maintainer will include it as config

Already said above that I would :)

Is bitmagnet running multiple simultaneous processes?

No

Is anyone having these issues with OpenWRT ALSO running through Gluetun VPN, and if not does that help at all?

Anyone tried this yet? Running through Gluetun should reduce the number of connections from thousands to 1.

Also is anyone having issues in this thread NOT running OpenWRT (which seems like the real issue here)?

[samwathegreat]

Do we expect a connections limit setting to be included in a future version of bitmagnet, or is the solution considered the correct way of dealing with the issue?

I really hope so the maintainer will include it as config

Already said above that I would :)

Is bitmagnet running multiple simultaneous processes?

No

Is anyone having these issues with OpenWRT ALSO running through Gluetun VPN, and if not does that help at all?

Anyone tried this yet? Also is anyone having issues in this thread NOT running OpenWRT (which seems like the real issue here)?

Fantastic on the future option to limit connections!

Very confused as to why I'm ending up with so many connections even with the prlimits...

I am running this on openwrt through NordVPN... The openwrt VM connects to NordVPN via OpenVPN protocol and tunnels all the traffic through. Still getting the very high number of connections.

[mgdigital]

I am running this on openwrt through NordVPN

Any chance you could try it through Gluetun and see if any improvement?

As per the example here:

https://github.com/bitmagnet-io/bitmagnet/blob/f4e4600d516a5d013b1a3c9b88ef0c4036228843/docker-compose.yml#L4-L80

[samwathegreat]

I am running this on openwrt through NordVPN

Any chance you could try it through Gluetun and see if any improvement?

Unfortunately, I don't use docker at all. And I'm not familiar with Gluetun. I run bitmagnet in LXC.

[mgdigital]

I am running this on openwrt through NordVPN

Any chance you could try it through Gluetun and see if any improvement?

Unfortunately, I don't use docker at all. And I'm not familiar with Gluetun. I run bitmagnet in LXC.

Aha, I've not used LXC before but AFAIK Gluetun will work on there too https://github.com/qdm12/gluetun/discussions/1482