libtorrent icon indicating copy to clipboard operation
libtorrent copied to clipboard

improve route selection for uTP sockets

Open arvidn opened this issue 4 years ago • 72 comments

libtorrent version (or branch): RC_1_2

This issue was posted here: https://github.com/qbittorrent/qBittorrent/issues/13094

The logic that picks which UDP socket to use for a uTP connection is simple and does not pick the correct one in some circumstances. At least make it take into account:

  • source hint
  • metric

The code is here:

std::vector<std::shared_ptr<listen_socket_t>> with_gateways;
std::shared_ptr<listen_socket_t> match;
for (auto& ls : m_listen_sockets)
{
	if (is_v4(ls->local_endpoint) != remote_address.is_v4()) continue;
	if (ls->ssl != ssl) continue;
	if (!(ls->flags & listen_socket_t::local_network))
		with_gateways.push_back(ls);

	if (match_addr_mask(ls->local_endpoint.address(), remote_address, ls->netmask))
	{
		// is this better than the previous match?
		match = ls;
	}
}
if (!match && !with_gateways.empty())
	match = with_gateways[random(std::uint32_t(with_gateways.size() - 1))];

if (match)
{
	utp_init_socket(impl, match);
	return match->local_endpoint;
}
ec.assign(boost::system::errc::not_supported, generic_category());
return {};

For both UDP and TCP connections, it seems interfaces that are DOWN are still considered.

arvidn avatar Jul 06 '20 06:07 arvidn

hmm.. I think it should still respect the fib.

What's "the fib"?

Maybe - on startup and interface changes - it should create one socket foreach interface.

That's how it works, yes. Unless you configure specific interfaces to use.

How do you get notified from the kernel about packets beeing ready? that is not select() as for tcp-sockets, or is it?

What do you mean by "packets being ready"? uTP runs over a single UDP socket, which means you can stick that in epoll/kqueue etc.

if you disable utp, and only use TCP connections, does everything work as expected?

I thought so... yes, but now deluge starts to do weird stuff even with tcp. It's doing the same thing, cycling through all possible ips/interfaces for src-ip but it looks like it's at least respecting the routingtable.

That's intentional, for load balancing. Unless you select a specific (set of) interfaces to use.

Even after a restart (so no fib/routingtable changes after starting the client) it still tries the ip of

a down interface the hosts public ip but none of the tunnel ips (yes, it's trying all src ips of the host, even for tcp....) this time it seems to respect the routing-table (at least). i disabled ecmp for easier debugging...

The log may have some details about why it didn't use the tunnel IPs. It's not supposed to use down interfaces, that's a problem.

arvidn avatar Jul 06 '20 17:07 arvidn

Hi @arvidn,

hmm.. I think it should still respect the fib. What's "the fib"?

Forwarding Information Base. It's basically the condensed and/or optimized form of the routingtable that used all sources (if there are routing protocols in the game) and did some basic optimizations. It's basically what the kernel works with; think of it as a compiled and "runtime" version of iptables/netfilter rules compared to the rules you type in.

What do you mean by "packets being ready"? uTP runs over a single UDP socket, which means you can stick that in epoll/kqueue etc.

so is there a callback whenever a packet is ready/buffered or do you have to actively select/poll or just busy-wait in a thread asking the socket if it has data?

anywho... let's start with tcp as this should be easier to debug, then tackle the connectionless-stuff.

I thought so... yes, but now deluge starts to do weird stuff even with tcp. It's doing the same thing, cycling through all possible ips/interfaces for src-ip but it looks like it's at least respecting the routingtable.

That's intentional, for load balancing. Unless you select a specific (set of) interfaces to use.

uff.. i think you can leave it out or better you should leave that out. it does not work in the majority of cases

  1. if you've one interface it doesnt matter or apply
  2. if you've two (the more common case), one is some rfc (wlog 192.168.0.254/24) net and one is routable.(wlog 1.2.3.4/31) It doesnt make sense to to ask into the internal net from 1.2.3.4 and neither in the other direction, from 192.168.0.254 via the gw 1.2.3.5; usually already 1.2.3.5 will drop the packet as coming from an rfc-network
  3. even if it is for loadbalancing, it can't use the wrong src-ip from the wrong net. If your provider allows you to spoof your src-address, usually rp (reverse-path) filtering will drop the answer packet as it comes in trough a different interface that it was sent from. If you've real ecmp, you usually have a loopback-ip that's reachable from all connected networks and you'll have to run some sort of routingprotocol that "fixes" the nexthop.
  4. it's up to the routing to decide which gateway to use. According to the tcpdumps, it is using the correct interface, but cycling trough all the ips the machine has.
  5. this brings up more "information leakage" stuff up, when you expect libtorrent to use where routing points it to and it's always "publishing" all ips the box owns. Firstly it was thought, this problem is the same as the vpn-problem where suddenly the "wrong" ip-address showed up. This is something else. a good way to avoid all the hacky "emergency stop" switches vpn clients now offer could be delt with by having no default-route and having the vpn add one (or two /1 like openvpn to be a "more specific"); vpn goes down, no routes there, no traffic, no leakage.

The log may have some details about why it didn't use the tunnel IPs. It's not supposed to use down interfaces, that's a problem.

can you give me a bump in the right direction how i could do this in a reproducible way?

i'm running deluge with "-L info", will bump this up to "debug" and see what it spits out, but maybe an example with https://github.com/arvidn/libtorrent/blob/RC_1_2/examples/client_test.cpp (recommended by @FranciscoPombal ) would be preferred.

with what should i test, current master?

the data in the tcpdump you saw were from just forced-reannounce to a tracker, where it used the ip from a down interface.

if you could hint me at how you normally gdb (or if that's somewhere in the docs) libtorrent, i could try to dig into this. my gdb is a little rusty.

toasta avatar Jul 06 '20 20:07 toasta

k.. some small infos, after battling with deluge vs. python3.7/logging on a python3.8 machine....

22:10:33 [DEBUG   ][deluge.core.alertmanager      :130 ] add_torrent_alert: added torrent: someFile
22:10:33 [DEBUG   ][deluge.core.alertmanager      :130 ] torrent_added_alert: someFile added
22:10:34 [INFO    ][deluge.core.torrentmanager    :668 ] Torrent someFile from user "localclient" loaded
22:10:35 [DEBUG   ][deluge.core.alertmanager      :130 ] state_changed_alert: someFile: state changed to: finished
22:10:35 [DEBUG   ][deluge.core.alertmanager      :130 ] torrent_finished_alert: someFile torrent finished downloading
22:10:35 [DEBUG   ][deluge.core.alertmanager      :130 ] state_changed_alert: someFile: state changed to: seeding
22:10:35 [DEBUG   ][deluge.core.alertmanager      :130 ] torrent_checked_alert: someFile checked
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] cache_flushed_alert: someFile
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] torrent_resumed_alert: someFile resumed
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[127.0.0.1:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[301.302.303.66:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[10.123.123.2:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[1.2.3.4:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[10.8.0.4:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[10.8.1.14:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[10.8.2.3:54133] sending announce (started)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[127.0.0.1:54133] No route to host "" (1)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[10.123.123.2:54133] No route to host "" (1)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[10.8.0.4:54133] No route to host "" (1)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[10.8.1.14:54133] No route to host "" (1)
22:10:36 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[10.8.2.3:54133] No route to host "" (1)
22:10:37 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[127.0.0.1:54133] No route to host "" (1) [No route to host]
22:10:37 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[10.123.123.2:54133] No route to host "" (1) [No route to host]
22:10:37 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[10.8.0.4:54133] No route to host "" (1) [No route to host]
22:10:37 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[10.8.1.14:54133] No route to host "" (1) [No route to host]
22:10:37 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[10.8.2.3:54133] No route to host "" (1) [No route to host]
22:10:40 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (1)
22:10:40 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (1)
22:10:40 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (1) [Connection timed out]
22:10:40 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (1) [Connection timed out]
22:10:57 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[301.302.303.66:54133] sending announce (started)
22:10:57 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[1.2.3.4:54133] sending announce (started)
22:11:03 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (2)
22:11:03 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (2)
22:11:03 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (2) [Connection timed out]
22:11:03 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (2) [Connection timed out]
22:11:58 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[301.302.303.66:54133] sending announce (started)
22:11:58 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[1.2.3.4:54133] sending announce (started)
22:12:04 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (3)
22:12:04 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (3)
22:12:04 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (3) [Connection timed out]
22:12:04 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (3) [Connection timed out]
22:13:44 [DEBUG   ][deluge.core.alertmanager      :130 ] save_resume_data_alert: someFile resume data generated
22:14:01 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[301.302.303.66:54133] sending announce (started)
22:14:01 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[1.2.3.4:54133] sending announce (started)
22:14:07 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (4)
22:14:07 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (4)
22:14:07 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (4) [Connection timed out]
22:14:07 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (4) [Connection timed out]
22:17:32 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[301.302.303.66:54133] sending announce (started)
22:17:32 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_announce_alert: someFile (some.tracker)[1.2.3.4:54133] sending announce (started)
22:17:38 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (5)
22:17:38 [DEBUG   ][deluge.core.alertmanager      :130 ] tracker_error_alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (5)
22:17:38 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[301.302.303.66:54133] Connection timed out "" (5) [Connection timed out]
22:17:38 [DEBUG   ][deluge.core.torrentmanager    :1389] Tracker Error Alert: someFile (some.tracker)[1.2.3.4:54133] Connection timed out "" (5) [Connection timed out]

I still think it should'nt use all ips on the host, especially not localhost. the "no route to host" .... dunno. could you gimme a hint on where in the code this is done? There's a route, but maybe you ask with a source-address set so it's using "source routing" (not really, but maybe the kernel is smart enough to say there would be no route to somewhere if this is set as a src-address. again... telnet some.tracker directly from a shell works flawlessly. conntrack doesnt seem to be the culprit. It's just logging that it SYN_SENT the packet but that it is still unreplied. I flushed that, too.

all 10.8 and the routeable 301 address would have internet connectivity; route would point at tun0/10.8.0.4

i think it is putting all on the correct interface but never with the correct src-address..

do you filter out rfc ips from route selection?

bf@$ sudo tcpdump -ni tun0 "port 67535"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
22:34:06.945346 IP 301.302.303.66.43091 > a.tracker.67535: Flags [S], seq 1335683990, win 64240, options [mss 1460,sackOK,TS val 3763069241 ecr 0,nop,wscale 7], length 0
22:34:06.966171 IP 1.2.3.4.55893 > a.tracker.67535: Flags [S], seq 4077430598, win 64240, options [mss 1460,sackOK,TS val 3322969615 ecr 0,nop,wscale 7], length 0
22:34:23.607488 IP 301.302.303.66.38565 > a.tracker.67535: Flags [S], seq 1272057056, win 64240, options [mss 1460,sackOK,TS val 3763085903 ecr 0,nop,wscale 7], length 0
22:34:23.607509 IP 1.2.3.4.54481 > a.tracker.67535: Flags [S], seq 560804832, win 64240, options [mss 1460,sackOK,TS val 3322986256 ecr 0,nop,wscale 7], length 0
22:34:23.639164 IP 301.302.303.66.54461 > a.tracker.67535: Flags [S], seq 85040338, win 64240, options [mss 1460,sackOK,TS val 3763085935 ecr 0,nop,wscale 7], length 0
22:34:23.639210 IP 1.2.3.4.58069 > a.tracker.67535: Flags [S], seq 738266937, win 64240, options [mss 1460,sackOK,TS val 3322986288 ecr 0,nop,wscale 7], length 0
22:34:23.712333 IP 301.302.303.66.41039 > a.tracker.67535: Flags [S], seq 13952375, win 64240, options [mss 1460,sackOK,TS val 3763086008 ecr 0,nop,wscale 7], length 0
22:34:23.712364 IP 1.2.3.4.37609 > a.tracker.67535: Flags [S], seq 2371420821, win 64240, options [mss 1460,sackOK,TS val 3322986361 ecr 0,nop,wscale 7], length 0
22:34:23.783689 IP 301.302.303.66.53773 > a.tracker.67535: Flags [S], seq 682434888, win 64240, options [mss 1460,sackOK,TS val 3763086079 ecr 0,nop,wscale 7], length 0
22:34:24.013189 IP 301.302.303.66.42565 > a.tracker.67535: Flags [S], seq 3564389759, win 64240, options [mss 1460,sackOK,TS val 3763086309 ecr 0,nop,wscale 7], length 0
22:34:24.013210 IP 1.2.3.4.58863 > a.tracker.67535: Flags [S], seq 1301810561, win 64240, options [mss 1460,sackOK,TS val 3322986662 ecr 0,nop,wscale 7], length 0
22:34:24.013228 IP 301.302.303.66.34513 > a.tracker.67535: Flags [S], seq 3094314752, win 64240, options [mss 1460,sackOK,TS val 3763086309 ecr 0,nop,wscale 7], length 0
22:34:24.013243 IP 1.2.3.4.47965 > a.tracker.67535: Flags [S], seq 3394956145, win 64240, options [mss 1460,sackOK,TS val 3322986662 ecr 0,nop,wscale 7], length 0
22:34:24.013400 IP 301.302.303.66.49567 > a.tracker.67535: Flags [S], seq 3859586362, win 64240, options [mss 1460,sackOK,TS val 3763086309 ecr 0,nop,wscale 7], length 0
22:34:24.013425 IP 1.2.3.4.59705 > a.tracker.67535: Flags [S], seq 2551732708, win 64240, options [mss 1460,sackOK,TS val 3322986662 ecr 0,nop,wscale 7], length 0
22:34:24.013686 IP 301.302.303.66.33055 > a.tracker.67535: Flags [S], seq 874523040, win 64240, options [mss 1460,sackOK,TS val 3763086309 ecr 0,nop,wscale 7], length 0
22:34:24.013703 IP 1.2.3.4.46385 > a.tracker.67535: Flags [S], seq 3816680831, win 64240, options [mss 1460,sackOK,TS val 3322986662 ecr 0,nop,wscale 7], length 0
22:34:24.622781 IP 1.2.3.4.54481 > a.tracker.67535: Flags [S], seq 560804832, win 64240, options [mss 1460,sackOK,TS val 3322987271 ecr 0,nop,wscale 7], length 0
22:34:24.622799 IP 301.302.303.66.38565 > a.tracker.67535: Flags [S], seq 1272057056, win 64240, options [mss 1460,sackOK,TS val 3763086918 ecr 0,nop,wscale 7], length 0
22:34:24.654774 IP 1.2.3.4.58069 > a.tracker.67535: Flags [S], seq 738266937, win 64240, options [mss 1460,sackOK,TS val 3322987303 ecr 0,nop,wscale 7], length 0
22:34:24.654791 IP 301.302.303.66.54461 > a.tracker.67535: Flags [S], seq 85040338, win 64240, options [mss 1460,sackOK,TS val 3763086950 ecr 0,nop,wscale 7], length 0
22:34:24.718774 IP 1.2.3.4.37609 > a.tracker.67535: Flags [S], seq 2371420821, win 64240, options [mss 1460,sackOK,TS val 3322987367 ecr 0,nop,wscale 7], length 0
22:34:24.718792 IP 301.302.303.66.41039 > a.tracker.67535: Flags [S], seq 13952375, win 64240, options [mss 1460,sackOK,TS val 3763087014 ecr 0,nop,wscale 7], length 0
22:34:24.814778 IP 301.302.303.66.53773 > a.tracker.67535: Flags [S], seq 682434888, win 64240, options [mss 1460,sackOK,TS val 3763087110 ecr 0,nop,wscale 7], length 0
22:34:25.038786 IP 1.2.3.4.46385 > a.tracker.67535: Flags [S], seq 3816680831, win 64240, options [mss 1460,sackOK,TS val 3322987687 ecr 0,nop,wscale 7], length 0
22:34:25.038801 IP 301.302.303.66.33055 > a.tracker.67535: Flags [S], seq 874523040, win 64240, options [mss 1460,sackOK,TS val 3763087334 ecr 0,nop,wscale 7], length 0
22:34:25.038826 IP 1.2.3.4.59705 > a.tracker.67535: Flags [S], seq 2551732708, win 64240, options [mss 1460,sackOK,TS val 3322987687 ecr 0,nop,wscale 7], length 0
22:34:25.038829 IP 301.302.303.66.49567 > a.tracker.67535: Flags [S], seq 3859586362, win 64240, options [mss 1460,sackOK,TS val 3763087334 ecr 0,nop,wscale 7], length 0
22:34:25.038837 IP 1.2.3.4.47965 > a.tracker.67535: Flags [S], seq 3394956145, win 64240, options [mss 1460,sackOK,TS val 3322987687 ecr 0,nop,wscale 7], length 0
22:34:25.038841 IP 301.302.303.66.34513 > a.tracker.67535: Flags [S], seq 3094314752, win 64240, options [mss 1460,sackOK,TS val 3763087334 ecr 0,nop,wscale 7], length 0
22:34:25.038844 IP 1.2.3.4.58863 > a.tracker.67535: Flags [S], seq 1301810561, win 64240, options [mss 1460,sackOK,TS val 3322987687 ecr 0,nop,wscale 7], length 0
22:34:25.038846 IP 301.302.303.66.42565 > a.tracker.67535: Flags [S], seq 3564389759, win 64240, options [mss 1460,sackOK,TS val 3763087334 ecr 0,nop,wscale 7], length 0
22:34:26.638792 IP 301.302.303.66.38565 > a.tracker.67535: Flags [S], seq 1272057056, win 64240, options [mss 1460,sackOK,TS val 3763088934 ecr 0,nop,wscale 7], length 0
22:34:26.638810 IP 1.2.3.4.54481 > a.tracker.67535: Flags [S], seq 560804832, win 64240, options [mss 1460,sackOK,TS val 3322989287 ecr 0,nop,wscale 7], length 0
22:34:26.670778 IP 301.302.303.66.54461 > a.tracker.67535: Flags [S], seq 85040338, win 64240, options [mss 1460,sackOK,TS val 3763088966 ecr 0,nop,wscale 7], length 0
22:34:26.670799 IP 1.2.3.4.58069 > a.tracker.67535: Flags [S], seq 738266937, win 64240, options [mss 1460,sackOK,TS val 3322989319 ecr 0,nop,wscale 7], length 0
22:34:26.734781 IP 301.302.303.66.41039 > a.tracker.67535: Flags [S], seq 13952375, win 64240, options [mss 1460,sackOK,TS val 3763089030 ecr 0,nop,wscale 7], length 0
22:34:26.734801 IP 1.2.3.4.37609 > a.tracker.67535: Flags [S], seq 2371420821, win 64240, options [mss 1460,sackOK,TS val 3322989383 ecr 0,nop,wscale 7], length 0
22:34:26.830788 IP 301.302.303.66.53773 > a.tracker.67535: Flags [S], seq 682434888, win 64240, options [mss 1460,sackOK,TS val 3763089126 ecr 0,nop,wscale 7], length 0
22:34:27.054837 IP 301.302.303.66.42565 > a.tracker.67535: Flags [S], seq 3564389759, win 64240, options [mss 1460,sackOK,TS val 3763089350 ecr 0,nop,wscale 7], length 0
22:34:27.054857 IP 1.2.3.4.58863 > a.tracker.67535: Flags [S], seq 1301810561, win 64240, options [mss 1460,sackOK,TS val 3322989703 ecr 0,nop,wscale 7], length 0
22:34:27.054864 IP 301.302.303.66.34513 > a.tracker.67535: Flags [S], seq 3094314752, win 64240, options [mss 1460,sackOK,TS val 3763089350 ecr 0,nop,wscale 7], length 0
22:34:27.054868 IP 1.2.3.4.47965 > a.tracker.67535: Flags [S], seq 3394956145, win 64240, options [mss 1460,sackOK,TS val 3322989703 ecr 0,nop,wscale 7], length 0
22:34:27.054882 IP 301.302.303.66.49567 > a.tracker.67535: Flags [S], seq 3859586362, win 64240, options [mss 1460,sackOK,TS val 3763089350 ecr 0,nop,wscale 7], length 0
22:34:27.054891 IP 1.2.3.4.59705 > a.tracker.67535: Flags [S], seq 2551732708, win 64240, options [mss 1460,sackOK,TS val 3322989704 ecr 0,nop,wscale 7], length 0
22:34:27.054922 IP 301.302.303.66.33055 > a.tracker.67535: Flags [S], seq 874523040, win 64240, options [mss 1460,sackOK,TS val 3763089351 ecr 0,nop,wscale 7], length 0
22:34:27.054939 IP 1.2.3.4.46385 > a.tracker.67535: Flags [S], seq 3816680831, win 64240, options [mss 1460,sackOK,TS val 3322989704 ecr 0,nop,wscale 7], length 0



toasta avatar Jul 06 '20 20:07 toasta

so is there a callback whenever a packet is ready/buffered or do you have to actively select/poll or just busy-wait in a thread asking the socket if it has data?

the former. the socket is waited on in epoll, when it's readable packets are drained until the socket doesn't have any more readable packets. then it's put back in epoll. This is just how UDP sockets are handled in general, nothing special.

uff.. i think you can leave it out or better you should leave that out. it does not work in the majority of cases

I'll be more specific. The details are documented here: http://libtorrent.org/manual-ref.html#multi-homed-hosts for TCP, unless you sett outgoing_interfaces, outgoing peer connections (over TCP) will not bind to any local interface, and the kernel will decide which one it will be routed through.

If you set outgoing_interfaces (which I think only makes sense if you also configure listen_interfaces to one or some specific interfaces you want to use), then sockets will be bound, round-robin, to those interfaces.

Tracker announces are different. Then we attempt to announce from every active listen interface. i.e. every IP that we accept incoming connection on, we attempt to announce to the tracker. In the normal case there's only one interface that can actually route to the tracker anyway, so that's the one that gets announced.

if you've one interface it doesnt matter or apply

round-robin on one interface is the same as no round-robin.

if you've two (the more common case), one is some rfc (wlog 192.168.0.254/24) net and one is routable.(wlog 1.2.3.4/31) It doesnt make sense to to ask into the internal net from 1.2.3.4 and neither in the other direction, from 192.168.0.254 via the gw 1.2.3.5; usually already 1.2.3.5 will drop the packet as coming from an rfc-network

I would think the most common case would be to not configure outgoing_interfaces, or to configure it to exactly one interface (say, tun0). If outgoing_interfaces is not set, there is not binding to local IPs for outgoing connections. (again, uTP is a bit different in this regard).

even if it is for loadbalancing, it can't use the wrong src-ip from the wrong net. If your provider allows you to spoof your src-address, usually rp (reverse-path) filtering will drop the answer packet as it comes in trough a different interface that it was sent from. If you've real ecmp, you usually have a loopback-ip that's reachable from all connected networks and you'll have to run some sort of routingprotocol that "fixes" the nexthop.

I wouldn't expect this to happen with TCP connections. with outgoing_interfaces, if SO_BINDTODEVICE is used, then bind() would be called with one of the IPs of that interface as well. Do you see this happening with TCP? (the UDP case is the bug this ticket is referring to).

it's up to the routing to decide which gateway to use. According to the tcpdumps, it is using the correct interface, but cycling trough all the ips the machine has.

I would expect, if I bind an outgoing TCP socket to local address A, and try to connect() it to a peer that can only be reached via interface with address B, the kernel would fail with EHOSTUNREACH. Is that not the case? Would it actually send the packet via interface with address B but maintain the local IP a?

this brings up more "information leakage" stuff up, when you expect libtorrent to use where routing points it to and it's always "publishing" all ips the box owns. Firstly it was thought, this problem is the same as the vpn-problem where suddenly the "wrong" ip-address showed up. This is something else. a good way to avoid all the hacky "emergency stop" switches vpn clients now offer could be delt with by having no default-route and having the vpn add one (or two /1 like openvpn to be a "more specific"); vpn goes down, no routes there, no traffic, no leakage.

I'm not sure I understand what problem this is referring to. I understand there's an issue with UDP packets being sent from a socket bound to one interface but sent to an address that really should have used a different interface. That's the issue this ticket is referring to.

Are you saying there's another issue? Does it affect TCP?

arvidn avatar Jul 06 '20 20:07 arvidn

does this have anything to do with it? http_tracker_connection.cpp:272

  // endpoints is an in-out parameter
  void http_tracker_connection::on_filter(http_connection& c
    , std::vector<tcp::endpoint>& endpoints)
  {
    // filter all endpoints we cannot reach from this listen socket, which may
    // be all of them, in which case we should not announce this listen socket
    // to this tracker
    auto const ls = bind_socket();
    endpoints.erase(std::remove_if(endpoints.begin(), endpoints.end()
      , [&](tcp::endpoint const& ep) { return !ls.can_route(ep.address()); })
      , endpoints.end());

    if (endpoints.empty())
   {
      fail(error_code(boost::system::errc::host_unreachable, system_category()));
      return;
    }

or is this the embedded tracker or some sort of logic which "myip" should be inserted into the announce?

toasta avatar Jul 06 '20 20:07 toasta

with what should i test, current master?

It seems this issue affects 1.2.x, so RC_1_2 is the branch this should be fixed in.

the data in the tcpdump you saw were from just forced-reannounce to a tracker, where it used the ip from a down interface.

Ok. down interfaces should be added as something to fix. Was the interface up when the client started, and then brought down?

if you could hint me at how you normally gdb (or if that's somewhere in the docs) libtorrent, i could try to dig into this. my gdb is a little rusty.

I'm not sure what you mean by this. When you build client_test, you can do: b2 link=static -j$(nproc) stage_client_test to copy the client_test executable to the current directory. b <file>:<line> to set breakpoint, next to step forward, c to continue, r to restart.

printing the verbose log from client_test can also be really useful. -f <log-file> and --alert_mask=all are useful command line arguments to client_test.

arvidn avatar Jul 06 '20 20:07 arvidn

Are you saying there's another issue? Does it affect TCP?

yup.. just look at the tcpdump:

bf@$ sudo tcpdump -ni tun0 "port 67535"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
22:34:06.945346 IP 301.302.303.66.43091 > a.tracker.67535: Flags [S], seq 1335683990, win 64240, options [mss 1460,sackOK,TS val 3763069241 ecr 0,nop,wscale 7], length 0
22:34:06.966171 IP 1.2.3.4.55893 > a.tracker.67535: Flags [S], seq 4077430598, win 64240, options [mss 1460,sackOK,TS val 3322969615 ecr 0,nop,wscale 7], length 0

tun0 would have 10.8.something but the tracker announcement is sent from two wrong ips, even one from a down interface but this time respecting the kernels decision over which interface to route (which is different in the udp case).

this is deluge with no "incoming address" or "outgoing interface" set.

toasta avatar Jul 06 '20 20:07 toasta

also, trackers and peer connections are different. for trackers libtorrent attempts to announce all IPs that can reach the tracker. There is no filtering for special address ranges. But a network, say, 127.0.0.1/8 without a gateway cannot reach a tracker with an IP address outside of that network, so that IP is not announced to the tracker.

arvidn avatar Jul 06 '20 20:07 arvidn

Tracker announces are different. Then we attempt to announce from every active listen interface. i.e. every IP that we accept incoming connection on, we attempt to announce to the tracker. In the normal case there's only one interface that can actually route to the tracker anyway, so that's the one that gets announced.

i think you should narrow this down to "we only announce it to the interface the routing to this tracker points at".

I'll be more specific. The details are documented here: http://libtorrent.org/manual-ref.html#multi-homed-hosts

hmmk... that could explain why it's not using the "correct" address when bound to tun interfaces with rfc-ips:

An item that's considered to be "local network" will not be used to announce to trackers outside of that network. For example, 10.0.0.2:6881l is marked as "local network" and it will only be used as the source address announcing to a tracker if the tracker is also within the same local network (e.g. 10.0.0.0/8).

again, i tested with nothing specified in "listen address" or "interface".

would that mean that if i bind to "tun0" which has a rfc ip, no annoncements would be made? what about nat then?

toasta avatar Jul 06 '20 21:07 toasta

also, trackers and peer connections are different. for trackers libtorrent attempts to announce all IPs that can reach the tracker. There is no filtering for special address ranges. But a network, say, 127.0.0.1/8 without a gateway cannot reach a tracker with an IP address outside of that network, so that IP is not announced to the tracker.

Ok.. so discard all that, that's just the magic used to figure out the IP that should be in the L7 Packet to the tracker?

toasta avatar Jul 06 '20 21:07 toasta

i think you should narrow this down to "we only announce it to the interface the routing to this tracker points at".

The whole point of doing this is to support multi-homed machines, i.e. supporting IPv4/IPv6 dual stacks.

again, i tested with nothing specified in "listen address" or "interface".

I don't really know the details of how Deluge and qBT map their settings to libtorrent's listen_interfaces and outgoing_interfaces.

would that mean that if i bind to "tun0" which has a rfc ip, no annoncements would be made?

If there is no route out of that local network, then only trackers inside that local network are considered reachable. If there's a gateway, then all public IPs are considered reachable.

what about nat then?

Same thing.

Ok.. so discard all that, that's just the magic used to figure out the IP that should be in the L7 Packet to the tracker?

Not really, it's used to determine whether a listen interface (local IP) should be used as the local address in a tracker announce. Not the tracker announce, there's one announce per IP that should be advertised.

arvidn avatar Jul 06 '20 21:07 arvidn

what do you mean by "RFC IPs". Do you mean reserved IP ranges?

arvidn avatar Jul 06 '20 21:07 arvidn

rfc

what do you mean by "RFC IPs". Do you mean reserved IP ranges?

the rfc1918 ips, 10/8, 172.16/12, 192.168/16

toasta avatar Jul 06 '20 21:07 toasta

but i might have already figured out my problem:

docs say

If an IP address is the unspecified address (i.e. 0.0.0.0 or ::), libtorrent will enumerate Listen IP addresses that are automatically expanded by libtorrent have some special rules. They are all assumed to be restricted to be "local network" unless the following conditions are met:

[...] OR the routing table contains a default route with a gateway for the corresponding network. This bullet only applies when expanding from an unspecified IP address. When explicitly specifying a device, we don't need to find a route to treat it as external.

This section describes the logic for determining whether an address has a default route associated with it or not. This is only used for listen addresses that are expanded from either an unspecified listen address (0.0.0.0 or ::) or from a device name (e.g. eth0).

A network is considered having a default route if there is a default route with a matching egress network device name and address family.

ok... let's imagine we've a simple host set up with one interface, eth0

ip a add 192.168.0.123/24 dev eth0
ip r add default via 192.168.0.254

everything should be fine; it is a rfc ip BUT it has a defaultroute so it is fine to announce trough it.

if we do

ip r del default
ip r add 0.0.0.0/1 via 192.168.0.254
ip r add 128.0.0.0/1 via 192.168.0.254

which basically is the same as one /0 route, no announcements at all would be made?

I really think we should consider asking the kernel about routing and only if this does return "no route to host" take that interface out of the list of connections to this tracker to be made. otherwise just use the interface and make the announcement over that host, let the kernel do all the routing and ip selection.. We don't have any control about which ip shows up on the tracker anyhow.

the other quirks i'm experiencing with strange ips showing up as source-ips on the wrong interface might be a side-effect of not having a default-route... don't know if we want to fix that or fix it the right way by asking the routing-table?

toasta avatar Jul 06 '20 21:07 toasta

I was wondering if this could have something to do with the "information leak" i mentioned above and described in https://github.com/arvidn/libtorrent/issues/4803.

openvpn does this "two /1" routes thing to be "more specific" and attract all the traffic.

But libtorrent wouldnt see this interface as it has no defaultroute set and the defaultroute is still the same as in the "unprotected" state?

I'll try to make it a bit more clear what i mean host has eth0 with 192.168.0.123/24 with defaultgw 192.168.0.254 openvpn comes along, talking to a vpn server 3.4.5.6, it connects to 3.4.5.6, figures out credentials and stuff set's a hostroute 3.4.5.6/32 via oldDefaultGateway 192.168.0.254 to not interrupt connectivity for itself configures a tun0 with 10.8.1.1/24 p2p endpoint 10.8.1.254/24 and adds routes 0.0.0.0/1 gw 10.8.1.254 128.0.0.0/1 gw 10.8.1.254

libtorrents now sees eth0 and tun0. tracker to contact has 6.7.8.9. It discharges tun0 as it has 10.8.1/24 address and 6.7.8.9 is not in that /24 and there is no route with a 0 mask pointing at tun0. so it announces only over eth0.

the packet is crafted with the src address of eth0 (or 0.0.0.0) and handed over to the kernel. the kernel knows that 6.7.8.9 is in 0.0.0.0/1 and routes it over tun0. If the packet crafted has 0.0.0.0 as src, it substitutes the src with the ip of tun0?

is this correct so far?

toasta avatar Jul 06 '20 21:07 toasta

is this correct so far?

No, I don't think so. Any route that has a gateway, is assumed to be able to reach the internet. Any such routes will have an interface too, right? that they are sending their packets via. I would expect, in your scenario, that both 0.0.0.0/1 and 128.0.0.0/1 would have tun0 as their interface. So, any listen_interface configured on tun0 (or one of its IPs) will be considered able to reach the internet, and the tracker (assuming it has a global IP).

arvidn avatar Jul 08 '20 08:07 arvidn

No, I don't think so. Any route that has a gateway,

finally managed to get it to compile (just my own stupidity or unfamiliraty with b2)

tried to download ubuntu server with example_client:

[Jul 07 21:28:46] listening on 0.0.0.0:0 (device: ) failed: [enum_route] [TCP] Resource temporarily unavailable

seems to be caused by "address/port already in use", could be more "speaking".

[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[127.0.0.1:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[10.123.123.2:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[10.8.0.4:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[10.8.1.14:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[10.8.2.3:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[[::1]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[[fe80::5caa:67ff:feef:285e%eth0]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[[fe80::9276:1f71:4550:493d%tun0]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[[fe80::897a:1a81:6bad:6ebc%tun1]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://torrent.ubuntu.com/announce)[[fe80::a087:3bcd:43c1:26b7%tun2]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[127.0.0.1:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[10.123.123.2:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[10.8.0.4:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[10.8.1.14:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[10.8.2.3:6882] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[[::1]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[[fe80::5caa:67ff:feef:285e%eth0]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[[fe80::9276:1f71:4550:493d%tun0]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[[fe80::897a:1a81:6bad:6ebc%tun1]:6881] No route to host "" (1)
[Jul 07 21:28:47] ubuntu-20.04-live-server-amd64.iso (https://ipv6.torrent.ubuntu.com/announce)[[fe80::a087:3bcd:43c1:26b7%tun2]:6881] No route to host "" (1)

might be caused by the "unable to bind".

tried with ./client_test -f log1 -k -e 500 --alert_mask=all --listen_interfaces=:33445

yields

[Jul 08 12:04:38] update listen interfaces: :33445
[Jul 08 12:04:38] parsed listen interfaces count: 1, ifaces: :33445
[Jul 08 12:04:38] reopen listen sockets
[Jul 08 12:04:38] listening on 0.0.0.0:0 (device: ) failed: [enum_route] [TCP] Resource temporarily unavailable

seem to be really trying to bind to port "0".

how do i correctly specify the port? Thought i saw somehere that ":33445" would do the trick. couldnt find it in the settings:

bf@bf-spielwiese:~/git/libtorrent/examples$ ./client_test --list-settings | grep -i -e port -e listen
listen_interfaces=<string>
report_true_downloaded=<bool>
report_web_seed_downloads=<bool>
no_connect_privileged_ports=<bool>
support_share_mode=<bool>
support_merkle_torrents=<bool>
report_redundant_bytes=<bool>
listen_system_port_fallback=<bool>
announce_crypto_support=<bool>
outgoing_port=<int>
num_outgoing_ports=<int>
listen_queue_size=<int>
ssl_listen=<int>
max_retry_port_bind=<int>
proxy_port=<int>
i2p_port=<int>

0.0.0.0:33445 seems to work.

[Jul 08 12:09:58] start session
[Jul 08 12:09:58] version: 1.2.7.0 revision: 3ed6fa64f
[Jul 08 12:09:58] max-connections: 803 max-files: 1024
[Jul 08 12:09:58]  *** session thread init
[Jul 08 12:09:58]  done starting session
[Jul 08 12:09:58] update listen interfaces: 0.0.0.0:33445
[Jul 08 12:09:58] parsed listen interfaces count: 1, ifaces: 0.0.0.0:33445
[Jul 08 12:09:58] reopen listen sockets
[Jul 08 12:09:58] listening on 0.0.0.0:0 (device: ) failed: [enum_route] [TCP] Resource temporarily unavailable
[Jul 08 12:09:58] attempting to open listen socket to: 127.0.0.1:33445 on device: lo local-network accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: lo): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: lo): Operation not permitted
[Jul 08 12:09:58]  listening on: 127.0.0.1 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 333.444.555.66:33445 on device: eth0 accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: eth0): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: eth0): Operation not permitted
[Jul 08 12:09:58]  listening on: 333.444.555.66 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 10.123.123.2:33445 on device: eth0 local-network accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: eth0): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: eth0): Operation not permitted
[Jul 08 12:09:58]  listening on: 10.123.123.2 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 1.2.3.4:33445 on device: eth1 accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: eth1): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: eth1): Operation not permitted
[Jul 08 12:09:58]  listening on: 1.2.3.4 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 10.8.0.4:33445 on device: tun0 local-network accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: tun0): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: tun0): Operation not permitted
[Jul 08 12:09:58]  listening on: 10.8.0.4 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 10.8.1.14:33445 on device: tun1 local-network accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: tun1): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: tun1): Operation not permitted
[Jul 08 12:09:58]  listening on: 10.8.1.14 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] attempting to open listen socket to: 10.8.2.3:33445 on device: tun2 local-network accept-incoming expanded-ip 
[Jul 08 12:09:58] bind to device failed (device: tun2): Operation not permitted
[Jul 08 12:09:58] bind to device failed (device: tun2): Operation not permitted
[Jul 08 12:09:58]  listening on: 10.8.2.3 TCP port: 33445 UDP port: 33445
[Jul 08 12:09:58] successfully listening on [TCP] 127.0.0.1:33445
[Jul 08 12:09:58] successfully listening on [UDP] 127.0.0.1:33445
[Jul 08 12:09:58] successfully listening on [TCP] 333.444.555.66:33445
[Jul 08 12:09:58] successfully listening on [UDP] 333.444.555.66:33445
[Jul 08 12:09:58] successfully listening on [TCP] 10.123.123.2:33445
[Jul 08 12:09:58] successfully listening on [UDP] 10.123.123.2:33445
[Jul 08 12:09:58] successfully listening on [TCP] 1.2.3.4:33445
[Jul 08 12:09:58] successfully listening on [UDP] 1.2.3.4:33445
[Jul 08 12:09:58] successfully listening on [TCP] 10.8.0.4:33445
[Jul 08 12:09:58] successfully listening on [UDP] 10.8.0.4:33445
[Jul 08 12:09:58] successfully listening on [TCP] 10.8.1.14:33445
[Jul 08 12:09:58] successfully listening on [UDP] 10.8.1.14:33445
[Jul 08 12:09:58] successfully listening on [TCP] 10.8.2.3:33445
[Jul 08 12:09:58] successfully listening on [UDP] 10.8.2.3:33445

seems to start announcing to the tracker(s):

[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** announce: [ announce_to_all_tiers: 0 announce_to_all_trackers: 0 num_trackers: 2 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 127.0.0.1:33445 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER REQUEST "https://torrent.ubuntu.com/announce" event: started abort: 0 ssl: (nil) port: 33445 ssl-port: 0 fails: 0 upd: 0 ep: 127.0.0.1:33445
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** QUEUE_TRACKER_REQUEST [ listen_port: 33445 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER_REQUEST [ url: https://torrent.ubuntu.com/announce?info_hash=%c4O%93%1b%1a9%86%85%12B%d7U%d0%acF%e9%fa%3c%5d2&peer_id=-LT1270-HNyHTNYc)ys!&port=33445&uploaded=0&downloaded=0&left=952107008&corrupt=0&key=0FEEB5D4&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:33445 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER REQUEST "https://torrent.ubuntu.com/announce" event: started abort: 0 ssl: (nil) port: 33445 ssl-port: 0 fails: 0 upd: 0 ep: 333.444.555.66:33445
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** QUEUE_TRACKER_REQUEST [ listen_port: 33445 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER_REQUEST [ url: https://torrent.ubuntu.com/announce?info_hash=%c4O%93%1b%1a9%86%85%12B%d7U%d0%acF%e9%fa%3c%5d2&peer_id=-LT1270-HNyHTNYc)ys!&port=33445&uploaded=0&downloaded=0&left=952107008&corrupt=0&key=0FEEB5D4&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:33445 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER REQUEST "https://torrent.ubuntu.com/announce" event: started abort: 0 ssl: (nil) port: 33445 ssl-port: 0 fails: 0 upd: 0 ep: 10.123.123.2:33445
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** QUEUE_TRACKER_REQUEST [ listen_port: 33445 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER_REQUEST [ url: https://torrent.ubuntu.com/announce?info_hash=%c4O%93%1b%1a9%86%85%12B%d7U%d0%acF%e9%fa%3c%5d2&peer_id=-LT1270-HNyHTNYc)ys!&port=33445&uploaded=0&downloaded=0&left=952107008&corrupt=0&key=0FEEB5D4&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:33445 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: ==> TRACKER REQUEST "https://torrent.ubuntu.com/announce" event: started abort: 0 ssl: (nil) port: 33445 ssl-port: 0 fails: 0 upd: 0 ep: 1.2.3.4:33445
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** QUEUE_TRACKER_REQUEST [ listen_port: 33445 ]

seems to try:

[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** update_tracker_timer: [ announce_to_all_tiers: 0 announce_to_all_trackers: 0 num_trackers: 2 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (1) [ep: 127.0.0.1:33445 ] "https://ipv6.torrent.ubuntu.com/announce" [ found: 0 i->tier: 1 tier: 1 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (1) [ep: 10.123.123.2:33445 ] "https://ipv6.torrent.ubuntu.com/announce" [ found: 0 i->tier: 1 tier: 1 working: 1 fails: 0 limit: 0 upd: 1 ]

note the down interface 1.2.3.4 all come back with "no route to host"

[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:33445 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (1) [ep: 10.8.2.3:33445 ] "https://ipv6.torrent.ubuntu.com/announce" [ found: 0 i->tier: 1 tier: 1 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** update tracker timer: next_announce < now 0 m_waiting_tracker: 1 next_announce_in: 553280650
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** increment tracker fail count [ep: 10.8.2.3:33445 url: https://ipv6.torrent.ubuntu.com/announce 1]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** announce: [ announce_to_all_tiers: 0 announce_to_all_trackers: 0 num_trackers: 2 ]
[Jul 08 12:09:58] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:33445 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]

bf@bf-spielwiese:~/git/libtorrent/examples$ cat log1  | perl -pe 's/XXXX./333.444.555./g' | grep -i tracker\ error | wc -l
10

there would be a route:


bf@bf-spielwiese:~/git/libtorrent/examples$ nslookup torrent.ubuntu.com
Non-authoritative answer:
Name:	torrent.ubuntu.com
Address: 91.189.95.21

bf@bf-spielwiese:~/git/libtorrent/examples$ ip r get 91.189.95.21
91.189.95.21 dev tun1 src 10.8.1.14 uid 1000 
    cache 
bf@bf-spielwiese:~/git/libtorrent/examples$ 

could you try this on your machine? adding two /1 routes and then deleting the defaultroute? Maybe i'm doing something wrong....

if i had to guess, the "no route to host" comes from this:

  void http_tracker_connection::on_filter(http_connection& c
    , std::vector<tcp::endpoint>& endpoints)
  {
    // filter all endpoints we cannot reach from this listen socket, which may
    // be all of them, in which case we should not announce this listen socket
    // to this tracker
    auto const ls = bind_socket();
    endpoints.erase(std::remove_if(endpoints.begin(), endpoints.end()
      , [&](tcp::endpoint const& ep) { return !ls.can_route(ep.address()); })
      , endpoints.end());

    if (endpoints.empty())
    {
      fail(error_code(boost::system::errc::host_unreachable, system_category()));
      return;
    }

toasta avatar Jul 08 '20 10:07 toasta

regarding route selection and the unbound udp sockets... you said you're creating one socket foreach interface that shows up?

I think other udp services bind to * and just drop packets to dst-ips (local ips that is in this case for incoming packets) that are not configured to be listening (like ntp does).

the can_route functions should then be doing something like this (should be a "global" function, not a function in an interface c++ instance)

can_route=false
ifaceOut=getRoute("5.6.7.8").get_interface()
foreach(l in listeningSockets) // if we're listening only on one interface, this should be a one-entry array, if there are more than this list, if "*" than any that are discovered
{
if (l== ifaceOut) { can_route=true; break}
}

if(can_route) { return 0}
return E_NO_ROUTE_TO_HOST;

i found this library, which supposedly gives you access to the kernel routing table in a portable way https://github.com/dugsong/libdnet

there's a c non-library-snippet with rt_netlink just for linux: https://stackoverflow.com/questions/3288065/getting-gateway-to-use-for-a-given-ip-in-ansi-c and https://www.linuxjournal.com/article/7356 and reading and parsing /proc/net/route could be a solution to, but one will have to do the longest-prefix-match by foot.

i've basically no idea how to do that on windows. there should be getipforwardtable https://docs.microsoft.com/de-de/windows/win32/api/iphlpapi/nf-iphlpapi-getipforwardtable?redirectedfrom=MSDN getBestRoute https://docs.microsoft.com/de-de/windows/win32/api/iphlpapi/nf-iphlpapi-getbestroute?redirectedfrom=MSDN or even better getBestInterface (seems to do v4 only)

toasta avatar Jul 08 '20 10:07 toasta

this also gives access to the interfaces. and this gives access to the routing table.

You seem to have figured out that you can't omit the IP or interface name in listen_interfaces. But even in the case of using 0.0.0.0, which should work, you get:

[Jul 08 12:09:58] listening on 0.0.0.0:0 (device: ) failed: [enum_route] [TCP] Resource temporarily unavailable

that doesn't look right. it looks like libtorrent is failing to list the routing table. Do you see anything in strace indicating what's failing?

arvidn avatar Jul 08 '20 20:07 arvidn

this also gives access to the interfaces. and this gives access to the routing table.

oh cool. you already have that rt_netlink route enumeration?

enum_interfaces seems to work, breakpointed at the return ret of std::vector<ip_interface> enum_net_interfaces(io_service& ios, error_code& ec)

(gdb) p *(ret._M_impl._M_start)@1
$3 = {{interface_address = {type_ = boost::asio::ip::address::ipv4, ipv4_address_ = {addr_ = {s_addr = 16777343}}, ipv6_address_ = {
        addr_ = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, 
        scope_id_ = 0}}, netmask = {type_ = boost::asio::ip::address::ipv4, ipv4_address_ = {addr_ = {s_addr = 255}}, ipv6_address_ = {
        addr_ = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, 
        scope_id_ = 0}}, name = "lo", '\000' <repeats 61 times>, friendly_name = '\000' <repeats 127 times>, 
    description = '\000' <repeats 127 times>, preferred = true}}

OT: is there a neater way to access a c++ vector in gdb?

[Jul 08 12:09:58] listening on 0.0.0.0:0 (device: ) failed: [enum_route] [TCP] Resource temporarily unavailable that doesn't look right. it looks like libtorrent is failing to list the routing table. Do you see anything in strace indicating what's failing?

it seems nl_dump_request returns no routes:

Thread 2 "client_test" hit Breakpoint 1, libtorrent::enum_routes (ios=..., ec=...) at ../src/enum_net.cpp:824
824		{
(gdb) l
819						&& std::strcmp(r.name, device) == 0;
820				}) != routes.end();
821		}
822	
823		std::vector<ip_route> enum_routes(io_service& ios, error_code& ec)
824		{
825			std::vector<ip_route> ret;
826			TORRENT_UNUSED(ios);
827			ec.clear();
828	
(gdb) next
[Thread 0x7ffff4f92700 (LWP 1197536) exited]
825			std::vector<ip_route> ret;
(gdb) 
827			ec.clear();
(gdb) 
1181			int sock = ::socket(PF_ROUTE, SOCK_DGRAM, NETLINK_ROUTE);
(gdb) 
1182			if (sock < 0)
(gdb) 
1188			std::uint32_t seq = 0;
(gdb) 
1190			char msg[NL_BUFSIZE] = {};
(gdb) 
1191			nlmsghdr* nl_msg = reinterpret_cast<nlmsghdr*>(msg);
(gdb) 
1192			int len = nl_dump_request(sock, RTM_GETROUTE, seq++, AF_UNSPEC, msg, sizeof(rtmsg));
(gdb) 
1193			if (len < 0)
(gdb) 
1195				ec = error_code(errno, system_category());
(gdb) p len
$1 = -1
(gdb) p seq
$2 = 1
(gdb) p msg
$3 = "4\000\000\000\030\000\002\000\000\000\000\000\316}\342\300\002\020\000\000\376\003\375\001\000\000\000\000\b\000\017\000\376\000\000\000\b\000\001\000\000\000\000\000\b\000\004\000<\000\000\000,\000\000\000\030\000\002\000\000\000\000\000\316}\342\300\002\000\000\000\376\003\375\001\000\000\000\000\b\000\017\000\376\000\000\000\b\000\004\000<\000\000\000\064\000\000\000\030\000\002\000\000\000\000\000\316}\342\300\002\020\000\000\376\003\375\001\000\000\000\000\b\000\017\000\376\000\000\000\b\000\001\000\000\001\000\000\b\000\004\000=\000\000\000\064\000\000\000\030\000\002\000\000\000\000\000\316}\342\300\002\020\000\000\376\003\375\001\000\000\000\000\b\000\017\000\376\000\000\000\b\000\001\000\000\002\000\000\b\000\004\000>\000\000\000"...
(gdb) p sock
$4 = 8
(gdb) p ec
$5 = (libtorrent::error_code &) @0x7ffff57913a0: {val_ = 0, failed_ = false, 
  cat_ = 0x5555558061d0 <boost::system::detail::cat_holder<void>::system_category_instance>}
(gdb) p ios
$6 = (libtorrent::io_service &) @0x55555581d490: {<boost::asio::execution_context> = {<boost::asio::detail::noncopyable> = {<No data fields>}, service_registry_ = 0x55555581d9b0}, impl_ = @0x55555581fc50}

 

does this help?

toasta avatar Jul 08 '20 21:07 toasta

strace says:

1197903 socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 8
1197903 sendto(8, {{len=28, type=0x1a /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=0, pid=0}, "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"}, 28, 0, NULL, 0) = 28
1197903 getsockname(8, {sa_family=AF_NETLINK, nl_pid=-822213377, nl_groups=00000000}, [12]) = 0
1197903 recvfrom(8, [{{len=52, type=RTM_NEWROUTE, flags=NLM_F_MULTI, seq=0, pid=3472753919}, {rtm_family=AF_INET, rtm_dst_len=16, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN}, {{nla_len=8, nla_type=RTA_DST}, inet_addr("0.0.0.0")}, {{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("tun0")}]}, {{len=52, type=RTM_NEWROUTE, flags=NLM_F_MULTI, seq=0, pid=3472753919}, {rtm_family=AF_INET, rtm_dst_len=16, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN}, {{nla_len=8, nla_type=RTA_DST}, inet_addr("0.1.0.0")}, {{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("tun1")}]}, {{len=52, type=RTM_NEWROUTE, flags=NLM_F_MULTI, seq=0, pid=3472753919}, {rtm_family=AF_INET, rtm_dst_len=16, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_T
[...]
1197903 recvfrom(8, "", 0, 0, NULL, NULL) = 0
1197903 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 8
1197903 setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
1197903 setsockopt(8, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = -1 EPERM (Operation not permitted)
1197903 bind(8, {sa_family=AF_INET, sin_port=htons(33445), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
1197903 getsockname(8, {sa_family=AF_INET, sin_port=htons(33445), sin_addr=inet_addr("127.0.0.1")}, [28->16]) = 0
1197903 listen(8, 3000)                 = 0
1197903 socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) = 9
1197903 setsockopt(9, SOL_SOCKET, SO_BINDTODEVICE, "lo\0", 3) = -1 EPERM (Operation not permitted)

OT: is this intentional that it does an accept() on all interfaces right after listen()?

after the listen() setup, there's a lot of getsockname and the setsockopt for TOS, then it's doing another round of "getting routes"

1197903 socket(AF_NETLINK, SOCK_DGRAM, NETLINK_ROUTE) = 22
1197903 sendto(22, {{len=28, type=0x1a /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=0, pid=0}, "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"}, 28, 0, NULL, 0) = 28
1197903 getsockname(22, {sa_family=AF_NETLINK, nl_pid=-756701297, nl_groups=00000000}, [12]) = 0
1197903 recvfrom(22, [{{len=52, type=RTM_NEWROUTE, flags=NLM_F_MULTI, seq=0, pid=3538265999}, {rtm_family=AF_INET, rtm_dst_len=16, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN}, {{nla_len=8, nla_type=RTA_DST}, inet_addr("0.0.0.0")}, {{nla_len=8, nla_type=RTA_OIF}, if_nametoinde
[...]
_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN}, {{nla_len=8, nla_type=RTA_DST}, inet_addr("0.103.0.0")}, {{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("tun1")}]}, ...], 4448, 0, NULL, NULL) = 4448
1197903 recvfrom(22, "", 0, 0, NULL, NULL) = 0
1197903 socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) = 22

toasta avatar Jul 08 '20 21:07 toasta

tiny babystep:

it fails in

if ((NLMSG_OK(nl_hdr, read_len) == 0) || (nl_hdr->nlmsg_type == NLMSG_ERROR))

in read_nl_sock

toasta avatar Jul 09 '20 18:07 toasta

patched around in the code a bit and ignored zero-length route informations (dunno where they originate). Code seems a bit duplicated in read_nl_sock and then the conversion to routes in the routes-vector but that might be to support all the platforms.

My error seems to arise from the different ways you can use to add routes. there's ip r add 3.4.5.0/24 dev tun0 and ip r add 3.4.5.0/24 via 3.4.5.254 dev tun0

on the first case, the elements of the std::vector<ip_route> that is returned after obtaining routes would return "0.0.0.0" for all ".gateway". It would have access to the interface name but it seems this brakes the route-selection libtorrent currently uses.

I'll keep looking at that.

toasta avatar Jul 09 '20 22:07 toasta

My error seems to arise from the different ways you can use to add routes. there's ip r add 3.4.5.0/24 dev tun0 and ip r add 3.4.5.0/24 via 3.4.5.254 dev tun0

These do different things though, right? The first does not have a gateway, so it will be assumed to only be able to reach IPs on that local network. The second one, having a gateway is expected to be able to reach the internet via it.

on the first case, the elements of the std::vector<ip_route> that is returned after obtaining routes would return "0.0.0.0" for all ".gateway". It would have access to the interface name but it seems this brakes the route-selection libtorrent currently uses.

As far as I can tell, in the first case, the only way to be able to reach the internet is if the route is a point-to-point connection, which isn't very well supported in libtorrent. I'm open to suggestion on how to discover that.

arvidn avatar Jul 11 '20 07:07 arvidn

@toasta I really appreciate your help in improving this. It sounds so far I should:

  • make sure down interfaces are not considered
  • skip over 0 length route messages from the netlink socket, rather than treating them as errors
  • take source -hint into account when selecting outgoing UDP socket for uTP

arvidn avatar Jul 11 '20 07:07 arvidn

if recv() returns 0, it means end-of-file, and it would seem to be an error to get an EOF rather than an NLMSG_DONE message, indicating we received all messages

arvidn avatar Jul 11 '20 08:07 arvidn

actually, looking at your strace again, it really looks like the problem is that libtorrent uses a fixed size buffer to read the netlink response into, and your routing table is larger than that buffer (of 8kiB).

I think the most important fix would be to support larger routing tables, so I'll make that buffer adapt.

arvidn avatar Jul 11 '20 08:07 arvidn

@toasta could you please test this patch? https://github.com/arvidn/libtorrent/pull/4871

arvidn avatar Jul 11 '20 09:07 arvidn

Hi arvidn,

i saw the 8k limit and was wondering why that is but i thought this was something like " a single message" buffer.

Now i can confirm it's the number of routes.

i came up with a dummy test:

TORRENT_TEST(t1)
{

  lt::io_service ios;
  error_code ec;
	std::vector<ip_route> routes = enum_routes(ios, ec);

  long co = 0;
  for(auto r: routes)
  {
    fprintf(stderr, "[%ld] route %s mask %s via gateway %s dev %s ?? %s\n", co++
       , r.destination.to_string().c_str()
       , r.netmask.to_string().c_str()
       , r.gateway.to_string().c_str()
       , r.name
       , "??"
    );
    //printf("[%ld] %s\n", co++, r.address().to_string().c_str());
  }
  fprintf(stderr, "** GOT [%ld] routes\n", co);
	TEST_CHECK(co != 0);

then added routes like

seq 01 76 | while read a; do sudo ip r add 10.0.$a.0/24 dev lo; done

[132] route ff00:: mask ff00:: via gateway :: dev tun2 ?? ??
** GOT [133] routes
[test_bf.cpp.t1] ***PASS***
bf@bf-spielwiese:~$ ip r show | wc -l
97

and it fails when i add one more route:

bf@bf-spielwiese:~/git/libtorrent/test$ bin/test_bf.test/gcc-9/debug/crypto-openssl/dht-off/export-extra-on/i2p-off/picker-debugging-on/threading-multi/test_bf 
test: bin/test_bf.test/gcc-9/debug/crypto-openssl/dht-off/export-extra-on/i2p-off/picker-debugging-on/threading-multi/test_bf
cwd_prefix = "/home/bf/git/libtorrent/test/test_tmp_1902989_"
rnd = 1b4e1112
cwd: /home/bf/git/libtorrent/test/test_tmp_1902989_0
** GOT [0] routes
[test_bf.cpp.t1]


***** test_bf.cpp:56 "TEST_ERROR: check failed: "co != 0"" *****

toasta avatar Jul 12 '20 20:07 toasta

I cherry-picked 8e0cac36626f8f8c9de765776a53450b3ed47840 while it doesnt fail anymore returning 0 routes, it doesnt get all of them.

bf@bf-spielwiese:~$ ip r show | wc -l
65635

vs

[302] route 249.253.0.0 mask 255.255.0.0 via gateway 10.8.1.1 dev tun1 ?? ??
[303] route 249.254.0.0 mask 255.255.0.0 via gateway 10.8.2.1 dev tun2 ?? ??
[304] route 0.0.0.0 mask 255.255.0.0 via gateway 0.0.0.0 dev  ?? ??
** GOT [305] routes


[test_bf.cpp.t1] ***PASS***

is there something wrong with my test-code?

toasta avatar Jul 12 '20 20:07 toasta

there's a program called enum_if in the test directory of libtorrent. It's not built by default, but you can do:

b2 link=static stage_enum_if

and you can run enum_if and it will print local addresses and routes, as seen by libtorrent.

arvidn avatar Jul 12 '20 20:07 arvidn

there's a program called enum_if in the test directory of libtorrent. It's not built by default, but you can do: and you can run enum_if and it will print local addresses and routes, as seen by libtorrent.

hmm. seems i missed that. But that's basically the same as my test does. the stock enum_if does also not see all routes:

249.253.0.0       255.255.0.0       10.8.1.1                           1500   -                 tun1
249.254.0.0       255.255.0.0       10.8.2.1                           1500   -                 tun2
0.0.0.0           255.255.0.0       -                                  0      -                 
========= Interfaces =========
bf@bf-spielwiese:~/git/libtorrent/test$ ./enum_if | grep -B 999999 "^========= Interfaces" | wc -l
308

toasta avatar Jul 12 '20 20:07 toasta

could you expand on why you need the routingtable at all?

My idea would be:

if no bind interface/ip is given just do a normal tcp/udp socket to that address from 0.0.0.0, kernel should take care of the rest.

if bind interface(s) is/are given

ask the kernel if it has a route to that. check if returned iface is in $boundInterfaces or ask with oiface supplied.

if bind ip(s) is/are given:

ask the kernel if it has a route to that. check if returned route has src set and that src is in $listenIps (seems to work for routes added w/o src set, too:

bf@bf-spielwiese:~$ ip r add 255.253.1.1 dev tun1
bf@bf-spielwiese:~$ ip r show | grep 255.253.1.1
255.253.1.1 dev tun1 scope link 
bf@bf-spielwiese:~$ ip r get 255.253.1.1
255.253.1.1 dev tun1 src 10.8.1.14 uid 1000 
    cache 

i don't think you need to specify the from address while querying the routingtable; this is imho only for src routing.

https://github.com/shemminger/iproute2/blob/master/ip/iproute.c#L1914

querying for connected doesnt seem to work, but fibmatch would return "linkdown":

bf@bf-spielwiese:~$ ip r get 1.2.3.5 connected
1.2.3.5 dev eth1 src 1.2.3.4 uid 1000 
    cache 
1.2.3.5 dev eth1 src 1.2.3.4 uid 1000 
    cache 
bf@bf-spielwiese:~$ ip r get 1.2.3.5 fibmatch
1.2.3.0/24 dev eth1 proto kernel scope link src 1.2.3.4 linkdown 

you can even ask with ports set.

This would enable ecmp and all the bells and whistles routing currently gives...

This should work on windows with the aforementioned get_best_interface, too.

toasta avatar Jul 12 '20 20:07 toasta

if no bind interface/ip is given just do a normal tcp/udp socket to that address from 0.0.0.0, kernel should take care of the rest.

Each local address need its own DHT node, and each local address need to (potentially) be announced to the tracker. Whether the TCP listen socket that actually accepts incoming connections is bound to a specific interface or not is not really important, since you can filter based on the local address of the new socket. But for UDP sockets, I don't think you can know which local address a packet was sent to when you receive it, just which address it was sent from. So for UDP I think you need one socket per address.

arvidn avatar Jul 12 '20 21:07 arvidn

thx. i think now i slowly start to get the bigger picture.

as you quoted "if no bind interface/ip is given", do your explanations only apply to the BIND * case?

and each local address need to (potentially) be announced to the tracker.

so you would do n separate announces with n different "&ip=" values? Is this where the "can_route()" comes into play and what you described as "loadbalancing" earlier?

I'm still more concerned about outgoing and why it does only work for some ips.. but that might all be because of my bit larger-ish routingtable and the random selection of routes that get stored. I think i tried switching to a proxy connections and as all routing-checks are then disabled, everything works...

But for UDP sockets, I don't think

ah.. now some things make more sense. That's why you bind for each local_ip and record on the socket-"object" which local-ip this is? do you create sockets unconditionally and just drop traffic on them if they're not in bindInterfaces/bindIps

But that's incoming stuff, for outgoing i still think you need the routingtable and the kernel should be way faster giving you outiface, srcaddr and you can check that against a potentially supplied $bindInterface, $bindIP. And i think i didnt see a subscription to the netlink socket for routing changes or new interfaces popping up, so you'd have to poll them every so often... and outgoing still doesnt work for me, even with the grow-on-demand libtorrent internal routing-table-fix.

toasta avatar Jul 12 '20 21:07 toasta

So for UDP I think you need one socket per address.

did you play with the ip_recvif socket flag and using recvmsg instead of recvfrom?

toasta avatar Jul 12 '20 21:07 toasta

as you quoted "if no bind interface/ip is given", do your explanations only apply to the BIND * case?

No, in general, each IP address that can accept incoming packets/connections should have its own DHT node and be announced to the tracker separately. the listen_interfaces setting is a way to pick which IPs to use. e.g. even though you're multi-homed, you may only want to accept connections on one interface.

so you would do n separate announces with n different "&ip=" values?

Yes, except instead of &ip= values, the source address is the IP that's announced. i.e. the socket must be bound, locally, to the IP address we're announcing. trackers don't really blindly trust &ip= arguments, but they can trust the source address, since it had a 3-way handshake, it proved it's not spoofed.

Is this where the "can_route()" comes into play and what you described as "loadbalancing" earlier?

can_route() is an attempt to determine whether a listen address can reach a tracker or not. If it cannot, that interface won't even be attempted to be announced. I think this function is only used for tracker announces, and the socket is bound to the specific address that's being announced. perhaps can_reach() would be a better name, because I don't think it really takes the routing table into account much.

I think i tried switching to a proxy connections and as all routing-checks are then disabled, everything works...

Do you mean you used the proxy flag on the listen_interface_t object? That's meant for socks5 connections, since a socks5 connection is assumed to be able to reach the public internet.

do you create sockets unconditionally and just drop traffic on them if they're not in bindInterfaces/bindIps

Well, not quite. By default the interfaces 0.0.0.0 and [::] are expanded to all IPs matching those address families. But if you specify which IPs to listen on, only sockets for those are opened.

But that's incoming stuff, for outgoing i still think you need the routingtable and the kernel should be way faster giving you outiface, srcaddr and you can check that against a potentially supplied $bindInterface, $bindIP.

Yes, for peer connections, for sure, then the local interface is not important. For DHT, UDP trackers and HTTP tracker connections however, I think the routing table has to be ignored.

And i think i didnt see a subscription to the netlink socket for routing changes or new interfaces popping up, so you'd have to poll them every so often... and outgoing still doesnt work for me, even with the grow-on-demand libtorrent internal routing-table-fix.

Yes, the routing table is listed every now and then.

arvidn avatar Jul 13 '20 15:07 arvidn

quick update, even with the 60k routingtable loaded, it still can't reach the tracker. All come back with "no route to host" Do you want me to investigate this further or are you allready too annoyed?

[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 127.0.0.1:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 1 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 127.0.0.1:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.123.123.2:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.1.14:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.2.3:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [::1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::5caa:67ff:feef:285e%eth0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::9276:1f71:4550:493d%tun0]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::897a:1a81:6bad:6ebc%tun1]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: [fe80::a087:3bcd:43c1:26b7%tun2]:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker error: (113) No route to host 
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ i->tier: 0 tier: 2147483647 working: 1 limit: 0 can: 0 sent: 0 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 333.444.555.66:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 1.2.3.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]
[Jul 13 20:47:18] ubuntu-20.04-live-server-amd64.iso: *** tracker: (0) [ep: 10.8.0.4:6881 ] "https://torrent.ubuntu.com/announce" [ found: 0 i->tier: 0 tier: 0 working: 1 fails: 0 limit: 0 upd: 1 ]

toasta avatar Jul 13 '20 18:07 toasta

quick update, even with the 60k routingtable loaded, it still can't reach the tracker. All come back with "no route to host" Do you want me to investigate this further or are you allready too annoyed?

I definitely want to go to the bottom of this, and fix it for the next stable release (1.2.8). I might not be able to keep up with you, developing fixes though :)

arvidn avatar Jul 13 '20 19:07 arvidn

i patched this in

diff --git a/src/http_tracker_connection.cpp b/src/http_tracker_connection.cpp
index e96624353..90c61f853 100644
--- a/src/http_tracker_connection.cpp
+++ b/src/http_tracker_connection.cpp
@@ -276,6 +276,18 @@ namespace libtorrent {
                // be all of them, in which case we should not announce this listen socket
                // to this tracker
                auto const ls = bind_socket();
+
+    unsigned long co = 0;
+    for(auto &e : endpoints)
+    {
+      fprintf(stderr, "BF in (%s) ls(%s) endpoint[%lu](%s) can_route(%d)\n"
+        , __func__
+        , ls.get_local_endpoint().address().to_string().c_str()
+        , co++, e.address().to_string().c_str()
+        , ls.can_route(e.address())
+      );
+    }
+
                endpoints.erase(std::remove_if(endpoints.begin(), endpoints.end()
                        , [&](tcp::endpoint const& ep) { return !ls.can_route(ep.address()); })
                        , endpoints.end());

this is the output it produces

BF in (on_filter) ls(127.0.0.1) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(333.444.555.66) endpoint[0](91.189.95.21) can_route(1)
BF in (on_filter) ls(10.123.123.2) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(1.2.3.4) endpoint[0](91.189.95.21) can_route(1)
BF in (on_filter) ls(10.8.0.4) endpoint[0](91.189.95.21) can_route(1)
BF in (on_filter) ls(10.8.1.14) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(10.8.2.3) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(::1) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(fe80::5caa:67ff:feef:285e%eth0) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(fe80::9276:1f71:4550:493d%tun0) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(fe80::897a:1a81:6bad:6ebc%tun1) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(fe80::a087:3bcd:43c1:26b7%tun2) endpoint[0](91.189.95.21) can_route(0)
BF in (on_filter) ls(127.0.0.1) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(10.123.123.2) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(10.8.1.14) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(10.8.2.3) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(::1) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(fe80::5caa:67ff:feef:285e%eth0) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(fe80::9276:1f71:4550:493d%tun0) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(fe80::897a:1a81:6bad:6ebc%tun1) endpoint[0](2001:67c:1560:8008::14) can_route(0)
BF in (on_filter) ls(fe80::a087:3bcd:43c1:26b7%tun2) endpoint[0](2001:67c:1560:8008::14) can_route(0)

the v4 vs. v6 is clear; correct route would be

bf@bf-spielwiese:~/git/libtorrent/examples$ ip r get 91.189.95.21
91.189.95.21 via 10.8.1.1 dev tun1 src 10.8.1.14 uid 1000 
    cache 

since the v4 vs. v6 can_route checks return 0, 1 would mean it will attempt to announce this ip?

this leaves us with my real public ip, the down interface and a tunnel (but the wrong one, according to the routing table). really curious why it likes only one of the tunnel interfaces....

if i remove the route,

bf@bf-spielwiese:~/git/libtorrent/examples$ ip r get 91.189.95.21
RTNETLINK answers: Network is unreachable

it basically looks the same:

BF in (on_filter) ls(10.8.0.4) endpoint[0](91.189.95.21) can_route(1)
BF in (on_filter) ls(1.2.3.4) endpoint[0](91.189.95.21) can_route(1)
BF in (on_filter) ls(333.444.555.66) endpoint[0](91.189.95.21) can_route(1)

if i remove most of the routes, it marks the tunnel interface with can_route(0)... if i add a defaultroute via one of the tunnelinterfaces, it marks the correct one with can_route(1). i'll look at all the "has_default_route" stuff in the code...

toasta avatar Jul 13 '20 20:07 toasta

yeah, there is some special treatment of default routes that probably should be a bit more sophisticated. I should look into that. I think it may be part of the heuristic to determine if a local address should be announced from or not.

since the v4 vs. v6 can_route checks return 0, 1 would mean it will attempt to announce this ip?

Right. And if the address you would expect to be announced from has can_route(0), that would indicate there's a problem (as you point out)

arvidn avatar Jul 13 '20 20:07 arvidn

I think the routing table patch is ready to land.

I'm working on a patch to get more information from interfaces, specifically operational-status, to ignore interfaces that are down.

I think the next step would be to fix the has_default_route() function (and probably rename it) to instead do something like:

  • return true if any global IP can be routed via the interface
  • return true if the union of all routes to an interface amounts to all addresses (this would cover the case of having two /1 routes)

arvidn avatar Jul 14 '20 16:07 arvidn

* return true if _any_ global IP can be routed via the interface
* return true if the union of all routes to an interface amounts to all addresses (this would cover the case of having two /1 routes)

the last one would fail for a poor man's loadbalancing like routing 0/1 via eth0 and 128/1 via eth1

Do you want me to dig into asking the kernel for single routes instead of the whole routingtable? something in the realm of what i said in https://github.com/arvidn/libtorrent/issues/4850#issuecomment-657273283?

toasta avatar Jul 14 '20 18:07 toasta

the last one would fail for a poor man's loadbalancing like routing 0/1 via eth0 and 128/1 via eth1

true, so maybe alternative 1 would be better then. i.e. if there is a route whose prefix is a global address (i.e. not reserved for local or other kinds of networks).

Do you want me to dig into asking the kernel for single routes instead of the whole routingtable?

The has_default_route() isn't really so much about finding the route for a specific target address. It's more of a heuristic to determine if an interface or local address is used to reach the internet. To know whether we should have a DHT node running on it, and announce it to trackers.

arvidn avatar Jul 14 '20 18:07 arvidn

@toasta Here are all 3 changes (stacked on top of the previous PRs) https://github.com/arvidn/libtorrent/pull/4881

arvidn avatar Jul 15 '20 11:07 arvidn

return true if any global IP can be routed via the interface true, so maybe alternative 1 would be better then. i.e. if there is a route whose prefix is a global address (i.e. not reserved for local or other kinds of networks).

hmm. Something doesnt feel right with this. I'm sure i could construct a "doesnt work" example.. Something like "there's a dmz with routable ips". consider Firewall-Host with eth0 111.0.0.1/31 (p2p uplink to provider) eth1 112.0.0.0/24 (dmz) eth2 10.100.123.0/24 (internal network)

Now it would create dht nodes for eth{0,1} and consider eth1 to be "the internet" as well but it'll usually never reach something there. And in cases where you want internal torrent for e.g. software distribution like facebook did, i think it's really fine if the user has to supply the correct information about which interfaces to use regardless of what routing/ips/... say. or imagine if you only want torrent on eth{1,2} but not internet facing. I know you do sort of "if this tracker is in the same subnet, then announce to it", but there might be more intermediate networks necessary to reach it.

If you want to check "real" internet access, what about asking 8.8.8.8 for a silly dns query (like SOA record for ".") or - because it's easier - just do a tcp-connect to google.com over that interface. I don't think google would mind that. Doing that once on startup or on discovery of new interfaces or interface-state-changes would do. mozilla should have a "do i have internet access"-check, too.

toasta avatar Jul 15 '20 14:07 toasta

just some observations with the stock buntu20.04 library and deluge: (trying to get it to do at least something)

(i'm back to less than 30 routes, one default route only)

  • the down interface really confused it.
  • i removed it's ip address and now it behaves a bit more deterministic
  • setting the interface name to use for outgoing connections didnt work really well
  • using it's src-address as outgoing interface made things bit better
  • it still tries to reach the tracker over the non-default-route-having interface
  • some torrents (with the same tracker) fail to reach the tracker, others dont.
  • i think the last one is the "loadbalancing" (the random stuff if there's more than one gateway) i experienced earlier and/or in the first post to the qbittorrent git-issue.

toasta avatar Jul 15 '20 14:07 toasta

Now it would create dht nodes for eth{0,1} and consider eth1 to be "the internet" as well but it'll usually never reach something there.

I don't think it's a big deal to try to announce that internal network to the tracker. If it can't reach it, no harm done.

And in cases where you want internal torrent for e.g. software distribution like facebook did, i think it's really fine if the user has to supply the correct information about which interfaces to use regardless of what routing/ips/... say. or imagine if you only want torrent on eth{1,2} but not internet facing. I know you do sort of "if this tracker is in the same subnet, then announce to it", but there might be more intermediate networks necessary to reach it.

As far as I understand, if there are other intermediate networks you must have a gateway configured, or it must be a point-to-point interface. Is that right?

does #4881 solve your problem of announcing tun0? I would expect it to solve everything except it may still send UDP packets for peer connections (uTP) over the wrong interface.

arvidn avatar Jul 15 '20 16:07 arvidn

i compiled has_default_route 5bfa9fe9c and brought deluge to run with it. (is that the right one?) I removed all listen/bind/... config and switched back to ecmp. Client-Connections seem to work, but all tracker announcements stay at "Announce sent", i dont see any of them put on the wire. 'll try other configurations...

toasta avatar Jul 15 '20 19:07 toasta

with one defaultroute it is sending out announces, but is still using all interfaces which returns "no route to host" and/or "timed out" most of the time (supposedly caused by the random stuff above)

toasta avatar Jul 15 '20 20:07 toasta

with "outgoing interface" set to tun0 and the defaultroute over tun0, it is still trying to announce over all interfaces. Some torrents somehow get stuck in "Announce sent", stopping and starting doesnt help.

Errors seem to cycle trough.. So some torrents (on the same tracker) are ok, some are in "Connection timed out", some in "no route to host" and some hard-stuck in "Announce sent".

clientconnections seem to work.

toasta avatar Jul 15 '20 20:07 toasta

I don't think it's a big deal to try to announce that internal network to the tracker. If it can't reach it, no harm done.

dunno. seems a bit dirty or at least "not pretty". As privacy concerns arose with failing vpns i'd avoid any announcements if they are not "configured" (as in 'use that interface only') or are doomed to fail anyway, like using wrong src-ips or something.

I know you do sort of "if this tracker is in the same subnet, then announce to it", but there might be more intermediate networks necessary to reach it.

As far as I understand, if there are other intermediate networks you must have a gateway configured, or it must be a point-to-point interface. Is that right?

there should be a route to it, but as far as i understood the "do i announce to that tracker over that interface"-logic, it'll only look for the netmask of the interface, not the actual routingtable.

[ Host-libtorrent 10.10.10.1/24] => [ 10.10.10.2/24 host1 10.20.20.2/24 ] <=> [10.20.20.1/24 host3 ]

host-libtorrent has a routes like

6.7.8.8/31 dev eth0
10.10.10.1/24 dev eth1
default via 6.7.8.9
10.20.20.0/24 via 10.10.10.2

if the tracker is 10.20.20.1 (host3), libtorrent will only see the defaultroute and not announce over eth1? I'm still voting for real routingtable queries :)

toasta avatar Jul 15 '20 20:07 toasta

with interface set to the ip of an interface and ~60k routes set, most of the torrents are stuck in "Announce sent" and some get trough. No Client-Traffic possible.

with all routes removed and one defaultroute via the ip of the interface configured in deluge, client-traffic possible but most of the tracker announcements go over the wrong interface or with the wrong srcaddress.

lots of torrents stuck in "Announce sent".

toasta avatar Jul 15 '20 21:07 toasta

with "outgoing interface" set to tun0 and the defaultroute over tun0, it is still trying to announce over all interfaces. Some torrents somehow get stuck in "Announce sent", stopping and starting doesnt help.

outgoing_interface is only for peer connections, not incoming connections. So in order to still accept incoming connections, the IP has to be announced to the tracker.

arvidn avatar Jul 15 '20 21:07 arvidn

dunno. seems a bit dirty or at least "not pretty". As privacy concerns arose with failing vpns i'd avoid any announcements if they are not "configured" (as in 'use that interface only') or are doomed to fail anyway, like using wrong src-ips or something.

Sure, it's not obvious to me that pinging google is any less dirty though.

arvidn avatar Jul 15 '20 21:07 arvidn

with "outgoing interface" set to tun0 and the defaultroute over tun0, it is still trying to announce over all interfaces. Some torrents somehow get stuck in "Announce sent", stopping and starting doesnt help.

outgoing_interface is only for peer connections, not incoming connections. So in order to still accept incoming connections, the IP has to be announced to the tracker.

I have no incoming connections, only outgoing. So "outgoing_interface" is not considered for tracker announcements?

toasta avatar Jul 15 '20 21:07 toasta

there should be a route to it, but as far as i understood the "do i announce to that tracker over that interface"-logic, it'll only look for the netmask of the interface, not the actual routingtable.

Right. Looking at the routing table wouldn't really help. The question isn't "If I contact this IP, will my connection go over this interface?". The question is: "Is it possible to accept incoming connections from the internet (or the tracker's network) on this interface".

The has_default_route patch makes the routing table check a bit more sophisticated, where it looks not only for a default route, but also if there's any route to that inerface, which seems like a reasonable indication.

arvidn avatar Jul 15 '20 21:07 arvidn

Sure, it's not obvious to me that pinging google is any less dirty though.

it proofs there is internet behind that interface and is less likely to leak any information. Dunno how udp tracker announcements actually work, but i suspect it'll fire-and-forget send a packet with the info_hash. a three-way handshake to google leaks basically no information..

Just trying to help and think how i would do a "is there internet" check.

toasta avatar Jul 15 '20 21:07 toasta

I have no incoming connections, only outgoing. So "outgoing_interface" is not considered for tracker announcements?

Correct. https://libtorrent.org/reference-Settings.html#outgoing_interfaces

arvidn avatar Jul 15 '20 21:07 arvidn

with all routes removed and one defaultroute via the ip of the interface configured in deluge, client-traffic possible but most of the tracker announcements go over the wrong interface or with the wrong srcaddress.

What does that mean exactly? Does it mean the default listen interfaces (which are 0.0.0.0:6881,[::]:6881) are expanded to interfaces that you wouldn't expect it to?

I think the bulk of this issue (or at least the first part of it that I'm trying to address) is concerning how the default settings (i.e. when the user doesn't specify listen IP(s) or interface(s)). When I say "expand" I mean turn 0.0.0.0 and [::] into specific IPs and interfaces. Which ones it expands to is printed to the log as "successfully listening on ..." (or something like that, assuming it succeeded in opening a listen socket).

The patch set fixes the following issues:

  1. the issue of failing to enumerate large routing tables (and also large interface lists for that matter)
  2. the issue of expanding the unspecified listen addresses to include interfaces that are DOWN.
  3. the issue of assuming an interface is a local-only network if there is no default route to it.

I would like to know your opinion about these changes.

  1. Do you think they are an improvement to the current behavior (I suppose (1) is a no-brainer probably)
  2. If so, do you think these patches work as described above?

lots of torrents stuck in "Announce sent".

Unfortunately I don't really know exactly what that means in Deluge.

arvidn avatar Jul 15 '20 21:07 arvidn

actually, I have one more question, apart from the issue of peer connections over uTP not honoring the routing table, can the other issues you experience be solved by the "escape hatch" of explicitly specifying the listen interface you want to use?

I think it's important that there is a manual override that works.

arvidn avatar Jul 15 '20 21:07 arvidn

So "outgoing_interface" is not considered for tracker announcements? Correct. https://libtorrent.org/reference-Settings.html#outgoing_interfaces

hmmk.. i read that but imho it's not clear, that tracker-announcements are completely separate.

This controls which IP address outgoing TCP connections are bound to,

there's no mentioning of "tracker" in the description of outgoing_interface, but listen_interfaces talks about outgoing (udp) trackers and that if you dont specify a port, outgoing tcp will still be possible. So i wrongly assumed, as the tracker-connection is a tcp connection, it'll adhere to that, too.

But you might consider changing that or adding another option so people having the "vpn dies and reveals my real ip" could simply put "tun0" in the outgoing_interface and never fear that any ip other than that of tun0 would be published.

toasta avatar Jul 15 '20 22:07 toasta

hmmk.. i read that but imho it's not clear, that tracker-announcements are completely separate.

Yeah, I agree that it's not all that clear. The mutli-homed section of the documentation should also describe some of the rationale we've talked about here, and how trackers, incoming peer connections and outgoing peer connections are treated differently (and really, only the outgoing peer connections take the routing table into account, or are meant to do that at least).

But you might consider changing that or adding another option so people having the "vpn dies and reveals my real ip" could simply put "tun0" in the outgoing_interface and never fear that any ip other than that of tun0 would be published.

If you put tun0 in both listen_interfaces and outgoing_interfaces you get that behavior (at least that's the intention).

I've been considering retiring the outgoing_interfaces in favor of a boolean setting to use the listen_interfaces as the outgoing interfaces (when enabled). Right now it's a bit weird that they can be set to separate interfaces.

arvidn avatar Jul 15 '20 23:07 arvidn

But you might consider changing that or adding another option so people having the "vpn dies and reveals my real ip" could simply put "tun0" in the outgoing_interface and never fear that any ip other than that of tun0 would be published.

If you put tun0 in both listen_interfaces and outgoing_interfaces you get that behavior (at least that's the intention).

tried that, i would expect it to not use any other interface for tracker-announcements, too, but this is what it does with both set to tun0

bf@bf-spielwiese:~$ ip a show dev tun1
    inet 10.8.1.5/24 brd 10.8.1.255 scope global tun1

listening on tun1, link-type RAW (Raw IP), capture size 262144 bytes
18:34:58.638808 IP 10.8.0.21.51429 > 333.333.333.333.333: Flags [S], seq 1217798281, win 64240, options [mss 1460,sackOK,TS val 175028227 ecr 0,nop,wscale 7], length 0
18:34:58.638831 IP 10.8.0.20.49271 > 333.333.333.333.333: Flags [S], seq 2428195436, win 64240, options [mss 1460,sackOK,TS val 2581027498 ecr 0,nop,wscale 7], length 0
18:34:58.638840 IP 10.123.123.2.52483 > 333.333.333.333.333: Flags [S], seq 2517015004, win 64240, options [mss 1460,sackOK,TS val 3134451541 ecr 0,nop,wscale 7], length 0
18:35:02.046570 IP 10.8.0.20.56215 > 333.333.333.333.333: Flags [S], seq 1326238818, win 64240, options [mss 1460,sackOK,TS val 2581030906 ecr 0,nop,wscale 7], length 0
18:35:03.054780 IP 10.8.0.20.56215 > 333.333.333.333.333: Flags [S], seq 1326238818, win 64240, options [mss 1460,sackOK,TS val 2581031914 ecr 0,nop,wscale 7], length 0
18:35:05.070774 IP 10.8.0.20.56215 > 333.333.333.333.333: Flags [S], seq 1326238818, win 64240, options [mss 1460,sackOK,TS val 2581033930 ecr 0,nop,wscale 7], length 0
18:35:06.830839 IP 10.123.123.2.52483 > 333.333.333.333.333: Flags [S], seq 2517015004, win 64240, options [mss 1460,sackOK,TS val 3134459733 ecr 0,nop,wscale 7], length 0
18:35:06.830860 IP 10.8.0.20.49271 > 333.333.333.333.333: Flags [S], seq 2428195436, win 64240, options [mss 1460,sackOK,TS val 2581035690 ecr 0,nop,wscale 7], length 0
18:35:06.830871 IP 10.8.0.21.51429 > 333.333.333.333.333: Flags [S], seq 1217798281, win 64240, options [mss 1460,sackOK,TS val 175036419 ecr 0,nop,wscale 7], length 0
18:35:09.134804 IP 10.8.0.20.56215 > 333.333.333.333.333: Flags [S], seq 1326238818, win 64240, options [mss 1460,sackOK,TS val 2581037994 ecr 0,nop,wscale 7], length 0

(note that tun1 has 10.8.1.5 and with both interface entries set to tun0 i would expect to see no bt traffic on tun1 and i would never expect traffic with the wrong src ip.

But you mentioned tracker announcements are totally different and dont adhere to any restrictions one would expect from setting interfaces? I think - at least for http-announcements, they should follow the users wish to only use this one interface.

supplying all 5 tunnels as outgoing and/or listen interfaces didnt make it better.

Putting dht aside...

  • if interface(s) are set, i still see no reason why it should http-announce to anywhere else
  • should not cycle trough the ips, especially not using routable src ips over rfc-interfaces and vice versa
  • if no interfaces set, not to/over any other interface the routing selects for that destination.

There might be a corner ultra special case where you want to use all available interfaces but always announce an ip from lo, but that's too special to think about right now and even if, announcing over one interface only should be enough.

and it's a little weird... since no tracker-announcements gets trough, i have no clients to connect to, altough those would use the correct interface?

toasta avatar Jul 18 '20 16:07 toasta

tried that, i would expect it to not use any other interface for tracker-announcements, too, but this is what it does with both set to tun0

What's the log output from libtorrent? I don't know what the log you posted means. libtorrent will print which interfaces and IPs it creates listen sockets for. I think that log would help understand what's going on.

You set listen_interfaces to tun0:<some port> and outgoing_interfaces to just tun0, is that right?

But you mentioned tracker announcements are totally different and dont adhere to any restrictions one would expect from setting interfaces?

No, I'm pretty sure I never said anything to suggest that. It's quite the opposite. tracker announces only care about the listen_interfaces setting, they don't care about your routing table (essentially).

supplying all 5 tunnels as outgoing and/or listen interfaces didnt make it better.

What did it do?

I'm getting the feeling that you may be getting error messages back from libtorrent, but you aren't seeing them because, maybe you're filtering alerts or not printing all of them.

Putting dht aside...

if interface(s) are set, i still see no reason why it should http-announce to anywhere else

neither do I

should not cycle trough the ips, especially not using routable src ips over rfc-interfaces and vice versa

Well, it still need to enumerate the IPs assigned to the specified interface, since there may be more than one. Say, one IPv4 and one IPv6 address.

if no interfaces set, not to/over any other interface the routing selects for that destination.

I would like to know more about what you mean here. There is no "destination" (at least not in deciding which IPs to "listen on", and announce to). The tracker itself is a destination, is that what you mean? do you mean that a tracker should always only be announced to once over whatever interface your routing table says it should use? (my understanding is that the routing table will always only have one best path to a destination IP).

That wouldn't work. Consider having one IPv4 and one IPv6 address, should only one be exposed to the tracker? Or having multiple internet connections, with separate global IPs, should only one be exposed to the tracker?

arvidn avatar Jul 18 '20 22:07 arvidn

What's the log output from libtorrent? I don't know what the log you posted means. libtorrent will print which interfaces and IPs it creates listen sockets for. I think that log would help understand what's going on.

The tcpdump should just show that it's using almost all source ips the box has, but it always uses the same outgoing interface which will only work if one uses the correct source ip.

No, I'm pretty sure I never said anything to suggest that. It's quite the opposite. tracker announces only care about the listen_interfaces setting, they don't care about your routing table (essentially).

k. then i got

also, trackers and peer connections are different. for trackers libtorrent attempts to announce all IPs that can reach the tracker.

wrong.

should not cycle trough the ips, especially not using routable src ips over rfc-interfaces and vice versa

Well, it still need to enumerate the IPs assigned to the specified interface, since there may be more than one. Say, one IPv4 and one IPv6 address.

yep, but it does this for all interfaces, not just the one that can actually reach the tracker (that's why i was always recommending the routing table)

if no interfaces set, not to/over any other interface the routing selects for that destination.

I would like to know more about what you mean here. There is no "destination" (at least not in deciding which IPs to "listen on", and announce to). The tracker itself is a destination, is that what you mean?

yup.

do you mean that a tracker should always only be announced to once over whatever interface your routing table says it should use? (my understanding is that the routing table will always only have one best path to a destination IP).

That wouldn't work. Consider having one IPv4 and one IPv6 address, should only one be exposed to the tracker?

nope. that's different address families. It's in two stacks and therefore like two different interfaces or two different running instances. I get that you need that, but you'll only do that if the address resolved has an AAAA record anyhow.

Let's agree that if we say "interface" we both mean the tuple {interface,adressfamily}, so {eth0,v4} and {eth0,v6} are two interfaces. And all ips of that single interface that is used to reach the destination (tracker) should be announced (putting mixing rfc and routable ips aside, that's a bonus) but only the ips of this interface the route points at. So if eth0 has 192.168.123.1 and 4.5.6.7, then it's usually pointless to use both to reach a tracker 11.12.13.14 as 192.168.123.1 will usually never be able to reach 11.12.13.14, but see below.

Or having multiple internet connections, with separate global IPs, should only one be exposed to the tracker?

that's a tough one. I think this has to be left to the user. You already hinted at "there's only one best route". This is usually true, putting ecmp and med aside. So if you announce both,

  • and the announcement really arrives there as you set a "wrong" srcip for the interface the announcement is routed over (one best route) (next-hop will usually drop that)
  • and if a connection really comes in via the interface this ip belongs to (usually it doesn't really get there, so above)
  • and If the return route does not point back over this interface
  • reverse path filtering will drop the packet.

What libtorrent - according to strace - does, is looping over all source addresses the box has with bind() - setting the src ip -, then with connect() to the address. The problem is that setting src doesn't make the connection stick to or select the right interface (i think you also describe that in the docs Binding an outgoing connection to a local IP does not necessarily make the connection via the associated NIC/Adapter.. So in my case, 90% of the packets sent out don't do anything and maybe delay everything for a very long time.

I think the problem is that torrents get stuck in "Announcements sent" state with this configuration. Skimming over the code i see no timer at first glance (but there is some) that puts a torrent out of this state.

force_reannounce has some ignore_min_interval reannounce_flags_t .

assuming that force_reannounce from torrent_handle.cpp is the function the application (deluge, qBitorrent) calls which queues the async call, maybe it's only qbittorent that does not set this flaq on "force reannounce" but it can't get to reannounce once it's in this state. Waiting very very long (hours) does sometimes make this state go away, Pausing and unpausing, not really.

This might also explain, why shutting down deluge/qBittorrent takes hours because it's waiting for all the "stopped" announcements to come back.

I don't know if this maybe is https://github.com/arvidn/libtorrent/blob/f4defa52b2d6bc3043e404229fdf8ec7c02db01e/src/torrent.cpp#L8994 combined with https://github.com/arvidn/libtorrent/blob/f4defa52b2d6bc3043e404229fdf8ec7c02db01e/src/torrent.cpp#L9012

and maybe the counter m_waiting_tracker (L9028)

What i also noticed in strace is that it's issuing the first n (5?) bind() calls followed by the connect() calls in sequence, then processes other stuff and then finally for the last (remaining?) src-ips (bind() and connect()) But that's only a observation, which likely does not have anything to do with the behavior.

Hope that helps...

toasta avatar Jul 19 '20 10:07 toasta

Sorry for butting in, and just quickly wanted to add that I get same behaviour as before when testing a fresh build of has_default_routes libtorrent branch(#4881) i.e. still both VPN and own IP are reported when checking announce-source-IPs from ipmagnet website(http-tracker) on windows 10 with two different provider's openvpn configs in deluge 2.0.3 and not binding any interfaces/ips, so default setup(config).

Unrelated, but I had never thought about if opening torrent client without any torrents in, and then starting VPN afterwards, then only the own IP is announced when adding ipmagnet link afterwards and not both, but that of-course also makes sence as having set listen/out sockets already. Qbittorrent's log states it sees the new IP and whatnot, but doesn't let libtorrent know seemingly to re-initialize. I know such behaviour is very exotic and doesn't make sence anyway lol, but just never thought about it, and saw it by chance when testing this and forgetting start VPN initially. This is both v1.2.7, v1.2.6 and has_default_routes i'm talking about here, but is torrent-client issue I understand.

Sorry again for butting in, and just added this comment because old ticket referrenced this ticket to be used instead.

Edit: Forgot to say that was thinking if the reason for the own IP isn't announced on Linux and only on windows(when not binding anything), when same profile used and same clients used both places from openvpn.net, maybe was because using tun on Linux and tap on windows, but changing to tap on Linux I couldn't get internet access, even though status from openvpn was OK, so couldn't test that scenario(I checked had nameservers in-place, but got no route to host on curl's/ping's etc.), and didn't know how use tun on windows, for additional testing neither.

mhertz avatar Jul 19 '20 10:07 mhertz

@toasta there are a few steps involved, and I realise some of the things I've said can easily be misinterpreted. I would like to clarify.

Step 1, to establish which IPs to listen on

This is determined by the listen_interfaces setting. There are 3 kinds of things it can contain:

  1. If listen_interfaces is configured to specific IPs, those IPs will be used for listen sockets to be bound to.
  2. If listen_interfaces is set to a device name, the IP addresses associated with that device are resolved and used. The name of the device is also recorded and used for SO_BINDTODEVICE. I use "device", "interface" and "adapter" interchangeably, and I don't refer it a pair of device-name + address family.
  3. If listen_interfaces is set to an unspecified address (which is the default, 0.0.0.0 and [::]) then all IP addresses associated with a local interface are enumerated and a heuristic is used to determine whether each IP can "reach the internet" or not. Basically, this is to prevent sending a tracker announce from 127.0.0.1. This is the only time the routing table is taken into account, for determining listening interfaces. The heuristics is described in detail in https://github.com/arvidn/libtorrent/pull/4881

I think option 3 here may be problematic in some cases. The set of patches in the pipeline to land attempts to improve this. For example by ignoring devices that are "down", and by not requiring a default route to an interface in order to consider it "connected to the internet" (to support two routes of 0.0.0.0/1 + 128.0.0.0/1).

Step 2, send announces to trackers

In this step, only IPs selected from step 1 are considered.

This is where the can_route() function is applied, but only to the IP addresses (also called listen_socket_t) we've already started listening on. Each such, listen_socket_t also has a netmask associated with it, pulled from the interface list. This netmask is used to determine whether the tracker is on the same network (not very likely) or if the listen_socket_t is a "local network" only. i.e. cannot reach the internet. Any interface that's explicitly configured is never considered "local".

If a listen_socket_t object is determined to be announced, there are two things that can happen.

  1. If the tracker is a UDP tracker, the UDP socket associated with the listen_socket_t is used to send the packets to the tracker. This UDP socket will have been bound (with bind() and possible with SO_BINDTODEVICE), which will determine the device that's used.

  2. If the tracker is an HTTP tracker, the TCP socket will be bound with bind() and possible SO_BINDTODEVICE before connecting.

Step 3, making peer connections

(let's assume outgoing_interfaces are not set). Outgoing TCP peer connections do not care about listen_socket_t objects at all. They are made just as normal. (if outgoing_interfaces are set, those IPs are used to bind() the outgoing sockets in a round-robin fashion, over all outgoing interfaces).

For outgoing uTP peer connections, one of the UDP sockets associated with a listen_socket_t will be used. This is because all uTP connections, UDP tracker connections and DHT messages share a UDP socket. This can lead to using the "wrong" interface for talking to a peer, relative to what the routing table says. I believe this is the other issue you are experiencing @toasta.

Let's agree that if we say "interface" we both mean the tuple {interface,adressfamily}, so {eth0,v4} and {eth0,v6} are two interfaces.

I don't think that's typically how the word "interface", "device" or "adapter" is used. I use those words to refer to a (physical or virtual) NIC. Each of which can be associated with any number of networks (i.e. address and prefix-length)

And all ips of that single interface that is used to reach the destination (tracker) should be announced (putting mixing rfc and routable ips aside, that's a bonus) but only the ips of this interface the route points at.

I'm not actually that familiar with how people would normally set up multiple networks over the same interface. But I've seen segmentations of local networks where some reach the internet, and some reach various internal services, that are fire-walled from the internet. Are you saying that it only ever makes sense to announce one network per interface?

I can imagine having multiple networks for an interface that still reach different egress points to the internet, in which case both would make sense to be announced.

Or having multiple internet connections, with separate global IPs, should only one be exposed to the tracker?

that's a tough one. I think this has to be left to the user. You already hinted at "there's only one best route". This is usually true, putting ecmp and med aside.

it is ultimately up to the user, because listen_interfaces can be configured.

What libtorrent - according to strace - does, is looping over all source addresses the box has with bind()

What do you mean by this exactly? libtorrent will enumerate the interfaces and IPs somewhat regularly. Is that what you see in strace? It's normally done via a netlink socket on linux though. I don't know what you mean by "with bind()", it's not calling bind() multiple times on the same socket with all the different local IPs, is it?

  • setting the src ip -, then with connect() to the address. The problem is that setting src doesn't make the connection stick to or select the right interface (i think you also describe that in the docs Binding an outgoing connection to a local IP does not necessarily make the connection via the associated NIC/Adapter.. So in my case, 90% of the packets sent out don't do anything and maybe delay everything for a very long time.

It sounds like the heuristic of picking which interfaces to "listen on" is mismatching your routing table, and attempts to bind to device fail (which isn't too surprising, since you need root privileges for that on linux). Hence the mismatch of source IP and interface.

Does that sound like it would explain the symptoms you see?

I would like to improve this situation and I'm open to suggestions. However, you say that even if you set listen_interfaces to the correct interface, it still doesn't work for you. Is that right? I would like to focus on that problem first, since that prevents people from getting it to work explicitly, even when circumventing the heuristic.

arvidn avatar Jul 19 '20 20:07 arvidn

@mhertz thanks for the report. Announcing both the VPN and the direct connection is a somewhat separate issue, and I think it deserves its own ticket, feel free to create a ticket for this issue (I seem to recall there is one in qbt, that probably should be linked). I'm open to suggestions for how to solve that. Maybe the default should be to only expand unspecified addresses to IPs from the "main" interface. It would require some heuristic to figure out which the main one is.

arvidn avatar Jul 20 '20 10:07 arvidn

Sorry Arvid, I guess I misunderstood your closing post in my ticket about this, like as if you had one of the commits in #4881 made for fixing this and to direct also conversation of this issue here.

I don't know enough about this to suggest if/what changes should possibly be made regarding that, and after reading your explenations previously I learned that it quite possibly was kinda unavoidable, given supporting multi-homed support, but anyway, I'll butt out again, and follow this interesting learningfull ticket from the outside :) Sorry again for the detraction.

mhertz avatar Jul 20 '20 11:07 mhertz

@toasta I believe the last issue I haven't addressed yet is the interface outgoing uTP peer connections are made over. I think the ideal solution would be to have a separate UDP socket, bound to the unspecified address to be used for peer connections.

The tricky thing with that is that SO_REUSEADDR and SO_REUSEPORT may have slightly different behavior on different platforms and incoming packets may still arrive on any socket. So for UDP trackers and the DHT nodes, they may receive packets on the "unspecified" socket, which would add some complication.

The simplest solution would probably be to take the routing table into account when picking the outgoing interface.

arvidn avatar Jul 21 '20 15:07 arvidn

@arvidn was this a duplicate of https://github.com/arvidn/libtorrent/issues/6772 or they're different issues?

summerqB avatar Jul 17 '22 08:07 summerqB