Download speed drops

[pummra]

I had seedsync set up as a Docker on my unRaid server. Everything seems to be set up right and working fine the download speeds maxing out my connection speed, however after a short while the download speed drops to like 900kb/s.

Looking at the log there is a ton of warnings of whatever is the active download file:

2020/12/28 10:53:17 - WARNING - ActiveScanner.ActiveScanner - Path does not exist:

And there is an error every minute or so:

2020/12/28 10:52:59 - ERROR - seedsync.Controller.Lftp - Lftp timeout exception Traceback (most recent call last): File "/app/python/lftp/lftp.py", line 128, in __run_command self.__process.expect(self.__expect_pattern, timeout=self._timeout) File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 344, in expect timeout, searchwindowsize, async) File "/usr/local/lib/python3.7/dist-packages/pexpect/spawnbase.py", line 372, in expect_list return exp.expect_loop(timeout) File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 181, in expect_loop return self.timeout(e) File "/usr/local/lib/python3.7/dist-packages/pexpect/expect.py", line 144, in timeout raise exc pexpect.exceptions.TIMEOUT: Timeout exceeded. <pexpect.pty_spawn.spawn object at 0x14d935caa190> command: /usr/bin/lftp args: ['/usr/bin/lftp', '-p', '22', '-u', 'XXXXX,', 'XXXXXX'] buffer (last 100 chars): b" BDRip x265 10bit EAC3 5.1 - ArcX)[TAoE].mkv' at 2340683994 (60%) 344.1K/s eta:4m [Receiving data]\r\n" before (last 100 chars): b" BDRip x265 10bit EAC3 5.1 - ArcX)[TAoE].mkv' at 2340683994 (60%) 344.1K/s eta:4m [Receiving data]\r\n" after: <class 'pexpect.exceptions.TIMEOUT'> match: None match_index: None exitstatus: None flag_eof: False pid: 344 child_fd: 6 closed: False timeout: 30 delimiter: <class 'pexpect.exceptions.EOF'> logfile: None logfile_read: None logfile_send: None maxread: 2000 ignorecase: False searchwindowsize: None delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 0.1 searcher: searcher_re: 0: re.compile(b'lftp XXXXX@XXXXX:.*>')

I have reduced my connections in settings to see if that would help. I have 1 maximum download with 6 connections total. The only other oddity in my setup is I have three Seedsync instances for different folders on the seedbox so I can more carefully manage what is synced etc.

[ipsingh06]

Hi @pummra . The first error is innocuous - it occurs right before a file starts downloading. The second error means that the LFTP process is not responding to Seedsync quickly enough. I would try running just a single Seedsync instance and see if you still get throughput drop-offs or these errors.

[pummra]

Will give it a go. I figured the warning wasn't much of an issue, however the log was spammed with the warnings. Like 100s of times for the same file as it was already being downloaded.

[quadcom]

One thing I've noticed, which makes sense, is if you leave SS in debug mode overall performance does suffer and download speeds are reduced slightly

[pummra]

Just to confirm I get the same issue even with one SeedSync instance without debug mode on.

[quadcom]

I've been seeing this happen after a while of heavy traffic. It's as if LFTP threads get stuck and won't connect. A restart doesn't always resolve the problem. Sometimes I have to resort to stopping the container and restarting it.

As I use Docker in a Synology NAS, I enabled controlled resources and gave it max CPU but limited RAM to 4GB. I've noticed that when these symptoms come up, RAM usage is pretty massive. In excess of 16GB.

I think Synology is a bit to blame for this as their Docker support is rather lacking.

I do have a significant log file. I'd rather not post here unredacted.

[ipsingh06]

@quadcom Next time you reproduce the issue, could you please send me a snapshot of what the system resources look like? Something like the top command output.

[quadcom]

OK, happened again with a large download queue.

I tried to get a TOP from the console in the docker container but I ran into another issue with LANG.

Resources reported to Synology Docker are well below overload or mem leaks territory. Copied a process list from within the Synology container.

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)                                                                  
I have no name!@df8098efbef4:/$           

[quadcom]

OK, happened again this time on the Ubuntu VM. On a restart, speeds kick up to 40+mbps but slowly drop down to approx 1.5mbps and stay there unless you restart the entire VM.

The CPU and LAN charts are from ESXi across ALL the VM's on the server. So it's not about resources getting maxed out somewhere else. This is def SS getting stuck somewhere.

Here's a screen recording - https://youtu.be/1FcqwDLqScc

For those that are tempted, don't bother hacking my box I've already changed the password.

We haven't heard from you for a while now. I hope everything is ok with you and your family.

seedsync.log

[ipsingh06]

Hey @quadcom, apologies for the delayed reply. There's definitely some throttling happening somewhere. I want to rule out that it's happening on the network side. Can you try running an LFTP mirror operation on the box via command-line? See if the same behaviour is observed without SeedSync in the mix.

To find an example of a mirror command to run, you can do this. Enable verbose logging in SeedSync by going to this URL: <gui IP address>:8800/server/config/set/general/verbose/True. (I should make this an option in Settings). Restart SeedSync and then queue something via the dashboard. Look for text in the logs Received command Action.QUEUE for file, and it should list the mirror command right after. Run this command after running lftp on the command-line on the same box where SeedSync runs. Observe the download speed of the operation with the jobs -v command.

[jamauai]

FYI, dropping the max parallel downloads/total connection numbers immediately fixed this issue. Here’s where I’m at for now. I’ll slowly increase until it breaks again to find where the limit is.

EDIT: I used to have it set to 32, 16, 16, 16, 16; which was probably way too much to begin with.

[quadcom]

In my exp,

Anything other than this and I get dropouts where the whole thing stops and restarts. Most of the time the restart doesn't get all the threads going so the overall speed doesn't hit the max it could hit. It mostly resolves itself the next file it starts but that can take a while as the DL speed can go from 40MB/s down to a few kb/s.

It's as if when multiple files are DL'ing at the same time, it overloads the system and can't keep up. Now, I'm running my SS in a VMWare Ubunutu server container with a quadcore and 4GB of RAM. So it can't be overloading that.