linkchecker
linkchecker copied to clipboard
Published
20 hours ago •
linkchecker
linkchecker
linkchecker hang with 0 links queued
The link checker finishes checking, but doesn't exit cleanly. I don't know how to replicate this, but here is the end of the log:
30 threads active, 24 links queued, 10163 links in 10464 URLs checked, runtime 39 minutes, 1 seconds
30 threads active, 5 links queued, 10183 links in 10471 URLs checked, runtime 39 minutes, 6 seconds
URL `plugin_upgrade/'
Name `docker plugin upgrade'
Parent URL https://docs.docker.com/v1.13/engine/reference/commandline/plugin/, line 352, col 11
Base https://docs.docker.com/v1.13/engine/reference/commandline/
Real URL https://docs.docker.com/v1.13/engine/reference/commandline/plugin_upgrade/
Check time 8.176 seconds
Result Error: 404 Not Found
URL `/v1.13/v1.12/'
Name `/v1.12/'
Parent URL https://docs.docker.com/v1.13/docsarchive/, line 314, col 37
Base https://docs.docker.com/v1.13/
Real URL https://docs.docker.com/v1.13/v1.12/
Check time 8.165 seconds
Result Error: 404 Not Found
10 threads active, 0 links queued, 10213 links in 10488 URLs checked, runtime 39 minutes, 11 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 16 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 21 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 26 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 31 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 36 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 41 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 46 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 51 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 39 minutes, 56 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 1 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 6 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 11 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 16 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 21 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 26 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 31 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 36 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 41 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 46 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 51 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 40 minutes, 56 seconds
1 thread active, 0 links queued, 10222 links in 10492 URLs checked, runtime 41 minutes, 1 seconds
It looked like it finished: Check time 8.176 seconds, but then it didn't actually exit.
I am running with 30 threads and perhaps that is too much. I know threading is hard. I've never seen this before. I let it run for an hour and 10 minutes before I killed the job. This is running off master.
indeed, 30 threads seems like a lot... did you try removing that setting?
of course, this is still a bug because we shouldn't just hang: we should fail more gracefully if we run out of resources, or whatever is happening thre.
I think I ran into this a second time with only 10 threads enabled. I didn't have status logging turned on so I can't be sure, but the check took 4 hours and the job finally aborted due to an absolute timeout. I've enabled status logging again in the meantime.
I'm having to use the master build instead of the 9.3.1 build you released, because I need to build it in a non-GUI environment and I can't seem to build 9.3.1 without the QT libraries. Installing them creates a big dependency burden that I don't want.
This is still happening with only 10 threads:
10 threads active, 1 link queued, 4764 links in 4962 URLs checked, runtime 12 minutes, 16 seconds
3 threads active, 0 links queued, 4776 links in 4969 URLs checked, runtime 12 minutes, 21 seconds
URL `http://www.aliyun.com/product/oss'
Name `Aliyun OSS'
Parent URL https://docs.docker.com/registry/storage-drivers/, line 186, col 85
Base https://docs.docker.com/registry/storage-drivers/
Real URL http://www.aliyun.com/product/oss
Check time 39.978 seconds
Size 258B
Result Error: ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 12 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 13 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 14 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 15 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 16 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 17 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 18 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 19 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 20 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 21 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 22 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 23 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 24 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 25 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 26 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 27 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 28 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 29 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 30 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 31 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 32 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 33 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 41 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 46 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 51 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 34 minutes, 56 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 1 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 6 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 11 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 16 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 21 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 26 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 31 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 36 seconds
2 threads active, 0 links queued, 4777 links in 4969 URLs checked, runtime 35 minutes, 41 seconds
I wonder if linkchecker is actually waiting for some HTTP connections that take forever to timeout. Can you check with netstat or lsof whether the hung linkchecker process has any open sockets?
Another idea for debugging this would be to set up a signal handler to use sys._current_frames() to dump the stack traces of all the threads, to see what those two threads are waiting for.
Speaking of which, I'd also like to use pstree or some other tool to verify that there really are threads running at this point, and it's not just an accounting error.
I've run into a similar issue when only using a single thread my link check is essentially stuck part way through the scan, current output is:
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 31 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 36 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 41 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 46 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 51 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 35 minutes, 56 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 1 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 6 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 11 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 16 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 21 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 26 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 31 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 36 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 41 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 46 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 51 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 36 minutes, 56 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 1 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 6 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 11 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 16 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 21 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 26 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 31 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 36 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 41 seconds
1 thread active, 1822 links queued, 1114 links in 3168 URLs checked, runtime 37 minutes, 46 seconds
It's been stuck like this for over 15 minutes now.
I don't know how to make sense of netstat but this is the output I get when looking for the linkchecker PID:
tcp 32 0 <my-server>:52010 159.89.204.151:https CLOSE_WAIT 13855/python
tcp 1 0 <my-server>:51854 45-126-129-126.sta:http CLOSE_WAIT 13855/python
tcp 1 0 <my-server>:39780 ip-119-47-116-232.:http CLOSE_WAIT 13855/python
tcp 0 0 <my-server>:54036 45-126-129-150.sta:http ESTABLISHED 13855/python
tcp 1 0 <my-server>:37384 ns3.telesmart.co.n:http CLOSE_WAIT 13855/python
tcp 1 0 <my-server>:35264 ip-112-109-69-201.:http CLOSE_WAIT 13855/python
tcp 32 0 <my-server>:33358 103.28.250.250.ip:https CLOSE_WAIT 13855/python
tcp 32 0 <my-server>:48522 104.27.134.77:https CLOSE_WAIT 13855/python
Here is (what I assume) is the relevant output from lsof:
linkcheck 13855 user 4u IPv4 929922756 0t0 TCP <my-server>:39780->ip-119-47-116-232.webhost.co.nz:http (CLOSE_WAIT)
linkcheck 13855 user 5u IPv4 929953929 0t0 TCP <my-server>:51854->45-126-129-126.static.wgtn.wheresmyserver.net.nz:http (CLOSE_WAIT)
linkcheck 13855 user 6u IPv4 929953935 0t0 TCP <my-server>:54036->45-126-129-150.static.wgtn.wheresmyserver.net.nz:http (ESTABLISHED)
linkcheck 13855 user 8u IPv4 929950525 0t0 TCP <my-server>:37384->ns3.telesmart.co.nz:http (CLOSE_WAIT)
linkcheck 13855 user 11u IPv4 929922761 0t0 TCP <my-server>:33358->103.28.250.250.ip.incapdns.net:https (CLOSE_WAIT)
linkcheck 13855 user 12u IPv4 929922767 0t0 TCP <my-server>:48522->104.27.134.77:https (CLOSE_WAIT)
linkcheck 13855 user 14u IPv4 929953233 0t0 TCP <my-server>:35264->ip-112-109-69-201.webhost.co.nz:http (CLOSE_WAIT)
linkcheck 13855 user 15u IPv4 929925882 0t0 TCP <my-server>:52010->159.89.204.151:https (CLOSE_WAIT)
I'm not really sure what I can do from here - I've used tcpdump to listen on the established connection port above but there doesn't appear to be any traffic.
Does this info help at all? Is there any other info I can dig out that would be useful?
I could reproduce this if you used stdin and there was an empty line in the file you feed it. When I was first testing something out I didn't catch that I had a trailing empty line:
% cat urls.txt
https://kodi.tv/sites/default/files/styles/hero_short/public/article/field_image/Android_Robot.png
http://ipv4.download.thinkbroadband.com/1GB.zip
https://kodi.tv/sites/default/files/styles/hero_short/public/article/field_image/84595474-00aa5100-ae50-11ea-86ab-cc7b4a47b977.png
http://ipv6.download.thinkbroadband.com:81/5MB.zip
https://www.dimensiondata.com/-/media/dimensiondata/nobglogo.png?la=en-GB&h=270&w=300&mw=350&hash=A835A2D6EEFEF2EAC25BEFA24E27BC66
http://www.ovh.net/files/100Mio.dat
[linkcheck] server:~/linkcheck% linkchecker --no-robots --no-warnings -o text --check-extern --stdin < urls.txt
LinkChecker 10.0.0.dev0 Copyright (C) 2000-2014 Bastian Kleineidam
LinkChecker comes with ABSOLUTELY NO WARRANTY!
This is free software, and you are welcome to redistribute it
under certain conditions. Look at the file `LICENSE' within this
distribution.
Get the newest version at https://linkchecker.github.io/linkchecker/
Write comments and bugs to https://github.com/linkchecker/linkchecker/issues
Start checking at 2020-07-07 17:58:06+000
1 thread active, 0 links queued, 5 links in 6 URLs checked, runtime 1 seconds
1 thread active, 0 links queued, 5 links in 6 URLs checked, runtime 6 seconds
1 thread active, 0 links queued, 5 links in 6 URLs checked, runtime 11 seconds
1 thread active, 0 links queued, 5 links in 6 URLs checked, runtime 16 seconds
1 thread active, 0 links queued, 5 links in 6 URLs checked, runtime 21 seconds
^CWARNING linkcheck.check 2020-07-07 17:58:28,586 MainThread interrupt; waiting for active threads to finish
WARNING linkcheck.check 2020-07-07 17:58:28,586 MainThread another interrupt will exit immediately
INFO linkcheck.check 2020-07-07 17:58:28,587 MainThread 1 URLs are still active. After a timeout of 5 minutes the active URLs will stop.
^CWARNING linkcheck.check 2020-07-07 17:58:29,482 MainThread user abort; force shutdown
I can't reproduce the empty trailing line as a cause. We have no other reports with v10. Would need testing with the latest version, and looking at the output from -Dall.
