desktop icon indicating copy to clipboard operation
desktop copied to clipboard

Massive slow down when NC tray icon is open

Open koppd opened this issue 3 years ago • 45 comments

⚠️ Before submitting, please verify the following: ⚠️

Bug description

When the tray icon is open (I don't know how to describe it) then the network performance is slow down by 90%. Here's an example of a 400 MB file transfer between two virtual machine with 1 GBit/s connection. => only 20 MBit/s grafik

Without the opened tray icon the speed is MUCH faster => 400 MBit/s grafik

Steps to reproduce

Copy some files (40 x 10 MB) and wait a second for the sync to start. (=upload to server; VFS is used) Open the tray icon a the speed goes down to 20 MBit/s; close the tray icon and the transfer speed accelerate to 400 Mbit/s

Expected behavior

Transfer should not be affected if tray icon is opened or not

Which files are affected by this bug

10485760 - Kopie (5) - Kopie - Kopie - Kopie - Kopie - Kopie - Kopie - Kopie - Kopie.txt

Operating system

Windows

Which version of the operating system you are running.

Windows 10

Package

Appimage

Nextcloud Server version

25.0.0 beta 5

Nextcloud Desktop Client version

3.6.0

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 3.3.6 to 3.4.0)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

Are you using an external user-backend?

  • [ ] Default internal user-backend
  • [ ] LDAP/ Active Directory
  • [ ] SSO - SAML
  • [ ] Other

Nextcloud Server logs

No response

Additional info

No response

koppd avatar Sep 09 '22 13:09 koppd

I'd like to add my current experience with a newly added sync folder using virtual files: When syncing 80000+ placeholders, I could see that progress on creating placeholder files is very fast in the beginning and slows down massively in the open tray over time. (sorry for being that vague, this is just an impression as I could not measure it).

Maybe this is just a problem of logging all the activity in the tray window, even while syncing initially? Or a repaint problem? It could maybe help to lazily load activity in initial or larger syncing runs.

Solving this heavy issue could also pay off in improving the speed problems with initial sync mentioned in https://github.com/nextcloud/desktop/issues/4424

I can confirm that restarting the client helps for some time until performance decreases again.

PhilippSchlesinger avatar Sep 14 '22 07:09 PhilippSchlesinger

I can also confirm this behavior with the placeholders (virtual files) in my computer with Windows 10.

wonx avatar Sep 14 '22 13:09 wonx

+1 same behavior noticed

CoYoNq avatar Sep 15 '22 11:09 CoYoNq

This issue should not be very hard to profile: e. g. add 100 000 small files (a maildir folder, for example) to a Nextcloud and measure the download to a Nextcloud Client. Even the upload to Nexcloud via a Client should suffer from the same problem.

PhilippSchlesinger avatar Sep 16 '22 12:09 PhilippSchlesinger

Hello I am on Version 3.6.2 grafik on Ubuntu 22.04.01 LTS grafik And experience the same. Sync never stops and the app freezes. Uses quite some CPU resources and the fan is constantly on. grafik

amilopowers avatar Nov 12 '22 21:11 amilopowers

In

https://help.nextcloud.com/t/windows-11-desktop-client-syncs-slower-after-a-while-and-syncs-1200-something-files/146347/7

it is mentioned that performance decreases with time and restarting the client helps, which underpins the suggestions I made in https://github.com/nextcloud/desktop/issues/4918#issuecomment-1246386007

I can confirm that restarting the client helps for some time until performance decreases again.

PhilippSchlesinger avatar Nov 15 '22 13:11 PhilippSchlesinger

See https://github.com/nextcloud/desktop/issues/4424#issuecomment-1341235591 for a good analysis of the issue.

PhilippSchlesinger avatar Dec 07 '22 21:12 PhilippSchlesinger

For me it doesn't appear to be opening the tray but if I open the settings page it grinds to a complete halt

The tray seems to work fine when started but over a few days it becomes completely bogged down, just opening the tray takes foverer, it refreshes very slowly, network transfer has ground to a halt

It seems like others that restarting the app every day keeps it somewhat alive but there's definitely some sort of issue with logs or something that block it up over time

Phil-Barker avatar Dec 20 '22 14:12 Phil-Barker

In my case it was beacause I set upload speed limitation to "automatic". I switched back to "unlimited" and everything works as it should.

amilopowers avatar Dec 20 '22 15:12 amilopowers

With latest Nextcloud Client 3.7.3 an inital sync on ~150k files took <1 hour where it was a whole night and endless errors in the past. Maybe you guys could also check again and see if it improved with the latest version.

Even if the sync process is now way faster, I can see a huge difference in speed where the NC tray window is shown or not. In my case there are multiple accounts connected in the Nextcloud Client. Switching to another accound in the tray window leads to faster progressing file sync in the account that currently does the inital sync. As previously mentioned in https://github.com/nextcloud/desktop/issues/4918#issuecomment-1246386007 this could be maybe due to a repaint problem.

PhilippSchlesinger avatar Feb 23 '23 12:02 PhilippSchlesinger

With latest Nextcloud Client 3.7.3 an inital sync on ~150k files took <1 hour where it was a whole night and endless errors in the past. Maybe you guys could also check again and see if it improved with the latest version.

Even if the sync process is now way faster, I can see a huge difference in speed where the NC tray window is shown or not. In my case there a multiple accounts connected in the Nextcloud Client. Switching to another accound in the tray window leads to faster progressing file sync in the account that currently does the inital sync. As previously mentioned in #4918 (comment) this could be maybe due to a repaint problem.

I will do a new timing test on my setup, I thought I saw a speed up after the last update, will verify it.

daramarak avatar Feb 23 '23 12:02 daramarak

It's still an issue with 3.7.3. I'm testing with a few big files and the sync speed is terrible slow when the tray window is open. When it's closed, it's fine. (up to 20 times faster)

Please keep in mind, that syncing thousands of small files are a different story and worth a different bug report. (which are surely exist) I guess it's a repaint issue.

koppd avatar Mar 04 '23 17:03 koppd

I am seeing the same issue. Windows 10. Latest client.

I'd even go a step further and say that the application syync speed is just MUCH slower than all other methods.

Downloading and uploading the same 12 GB file to the same location through 3 different methods (Samba, Web client, and windows application) showed the windows client is 10x+ slower than any other method, even in best case for windows app.

I have a 10GbE network, and expect to run up against my hard drive throughput limit on the server, not some arbitrary 25 MB/sec limit. The app is terribly slow.

comparison

jonofmac avatar Apr 14 '23 05:04 jonofmac

I am seeing the same issue. Windows 10. Latest client.

@jonofmac Did you also check if it makes a difference in the measurements if the tray window is shown or not?

PhilippSchlesinger avatar Apr 14 '23 10:04 PhilippSchlesinger

I am seeing the same issue. Windows 10. Latest client.

@jonofmac Did you also check if it makes a difference in the measurements if the tray window is shown or not?

These tests were all done with the tray icon closed. When i open the tray, the speeds plummet to 5-8 MB/sec. Close tray and it jumps back to 25

jonofmac avatar Apr 14 '23 15:04 jonofmac

The issue has some more implications than I can read from here: See the attached Gif:

  1. A big file is placed into a sync directory (BigFile2 with ~4.5GB)
  2. The Nextcloud (3.8.1) client is started
  3. The client scans the drive for a few seconds and starts the upload with 450MBit (~50% of the 1GBit connection)
  4. The tray icon window is opened - the upload comes to a crawl to 5-8 MBit
  5. The tray icon is closed and speed goes up to ~60MBit, but does not recover the original "full speed"
  6. The client is closed so the upload is cancelled
  7. The file is uploaded to the server using a Web client with the full GBit network limit.

Basically opening the client in any way destroys any upload performance. Additionally it never reaches its full network potential.

NextcloudClientBug

pampersrocker avatar May 17 '23 00:05 pampersrocker

I have the same issue that my nextcloud app not even responding on my windows 11 machine. I used the latest stable 3.8.2 for some days and now I downgraded to 3.7.3 and it seems to work well.

aortmannm avatar May 21 '23 08:05 aortmannm

This problem is affecting users on virtual files only it seems. We have the latest available version of Nextcloud 25, Nextcloud 25.0.6 on server and the clients have the latest client installed, version 3.8.2. We signed up a new company and they have about 6 users and the files shared among them totals more than 800 000 small files.

The users cannot work at all as their PC's are too slow. It is as if Nextcloud restarts the sync just as it completed the previous sync and just never stops.

This is a huge issue and we are about to loose our new customer. As this thread is almost a year old, it seems the Nextcloud devs are not looking into this or releasing any fixes at all. What is going on?

JohanBoshoffWebAware avatar May 24 '23 06:05 JohanBoshoffWebAware

I have the same issue. The reason is subfolders number. This software provide a continuous scan of sync folders. So, if your folder is pretty big and complex, everything start to be slow. The only way to "fix" this, is to make separated sync works for each folder, instead of syncing parent folder with everything inside.

StefanoV1989 avatar May 24 '23 08:05 StefanoV1989

This problem is affecting users on virtual files only it seems. We have the latest available version of Nextcloud 25, Nextcloud 25.0.6 on server and the clients have the latest client installed, version 3.8.2. We signed up a new company and they have about 6 users and the files shared among them totals more than 800 000 small files.

The users cannot work at all as their PC's are too slow. It is as if Nextcloud restarts the sync just as it completed the previous sync and just never stops.

This is a huge issue and we are about to loose our new customer. As this thread is almost a year old, it seems the Nextcloud devs are not looking into this or releasing any fixes at all. What is going on?

I can confirm that using the Nextcloud client 3.7.3 works well and the issue stops. Please Nextcloud, please fix the latest client to have better performance using sync.

JohanBoshoffWebAware avatar May 24 '23 10:05 JohanBoshoffWebAware

3.7.3 stopped working for me after some hours. I needed to disable now the virtual files.

aortmannm avatar May 24 '23 11:05 aortmannm

Windows client version 3.9, slowdown issue with tray open issue is still here.

D1StrX avatar Jul 01 '23 20:07 D1StrX

I can confirm the slowdown when the tray is open. My setup:

  • Windows 11 running Nextcloud Desktop Client 3.9.0
  • Server running 26.0.3
  • Both server and client use SSDs and are connected through 1Gb/s Ethernet

Test case:

  • Uploading a 2Gb file using the client
  • With / without VFS

Observations:

  • Initially, I get ~470 Mbps upload speed
  • opening the tray reduces the speed to 8 Mbps
  • closing it gives me 90 Mbps
  • For the rest of the transfer, it stays at 8 Mbps (tray open) or 90 Mbps (tray closed)

Findings:

  • Opening the tray makes the transfer massively slower
  • Closing it increases the transfer speed, but doesn't get back to the initial speed
  • I'm seeing this behavior for uploads and downloads
  • Using VFS or not doesn't make a difference, the slowdown is present in both cases

I also recorded a quick video showing the slowdown when downloading a large file (using VFS): https://www.youtube.com/watch?v=THqeszUO9Ws

In the video, the transfer already starts "medium slow". I'm not sure yet when exactly this happens. Restarting the client and starting the download without opening the tray resulted in a higher initial transfer speed.

I guess it's a repaint issue.

That's what I was thinking too...

I'll take a look at the code and see if I can find something.

felixwrt avatar Jul 14 '23 19:07 felixwrt

I've done some investigation and found what's causing the issue. While synchronizing, the window shows an animated blue sync icon. The animation of the icon is causing the slowdown. I've disabled the animation locally and now the synchronization runs with full speed - whether the tray is open or not.

That's the change that disables the animation:

diff --git a/src/gui/tray/syncstatussummary.cpp b/src/gui/tray/syncstatussummary.cpp
index 8f74985c8..adf8519cd 100644
--- a/src/gui/tray/syncstatussummary.cpp
+++ b/src/gui/tray/syncstatussummary.cpp
@@ -151,7 +151,7 @@ void SyncStatusSummary::setSyncStateForFolder(const Folder *folder)
         break;
     case SyncResult::SyncRunning:
     case SyncResult::NotYetStarted:
-        setSyncing(true);
+        setSyncing(false);
         setSyncStatusString(tr("Syncing"));
         setSyncStatusDetailString("");
         setSyncIcon(Theme::instance()->syncStatusRunning());

Here's a video showing the slowdown when the tray is open: https://drive.google.com/file/d/18FHSYBcek133wcvFyhY27Y7bwTN63GwM/view?usp=sharing

And here's the same upload operation again with the animation being deactivated: https://drive.google.com/file/d/1iNM3KMazQZAqZP_VrSuQ9cKzOUOyNmu9/view?usp=sharing

I'm not sure yet why the animation is causing such a big slowdown. CPU usage isn't high while this is happening. I suspect that it's a scheduling issue somehow (e.g. that scheduling the animation X times per second is in the same loop as the code driving the file transfer), but that's only a guess currently. I'll see if I can find the root cause.

Anyway, would it be an option to disable the animation for the time being? It's not as pretty, but the animation is causing slowdowns even after the tray has been closed again and I'd argue for performance rather than aesthetics.

I can do a PR deactivating the animation if that's a viable option.

felixwrt avatar Jul 17 '23 07:07 felixwrt

In my opinion the faster Nextcloud can operate the better and I am sure no one will mind the animation going away. Either the Nextcloud team must look into this issue and release a fix or the animation must be disabled completely until there is a fix

JohanBoshoffWebAware avatar Jul 17 '23 07:07 JohanBoshoffWebAware

The animation of the icon is causing the slowdown.

Great finding, kudos!

Can you see if this holds true for the animation in the settings/account window (like https://nextcloud.com/wp-content/uploads/2022/04/linux.png)?

PhilippSchlesinger avatar Jul 17 '23 08:07 PhilippSchlesinger

Can you see if this holds true for the animation in the settings/account window (like https://nextcloud.com/wp-content/uploads/2022/04/linux.png)?

The sync icon in the settings window isn't animated (at least for me) and therefore doesn't cause a slowdown.

felixwrt avatar Jul 17 '23 10:07 felixwrt

The sync icon in the settings window isn't animated (at least for me) and therefore doesn't cause a slowdown.

You are absolutely right about that. I find it quite suspicious that the

'x seconds left, x MB of x MB, file x of x'

information updates super fast and seems to be repainted multiple times a second. (in the tray and in the settings window) Maybe that slows down the client even more?

PhilippSchlesinger avatar Jul 24 '23 11:07 PhilippSchlesinger

Probably. The text is updated whenever there's an update on the synchronization progress. However, the string content is cached and the UI is only redrawn when the newly computed string is different from the one being shown currently.

A good next step would be to investigate how many of those progress messages are emitted per second. If there's no good reason for high update rates, the rate of these messages could be reduced to < 10 per second to see whether it does make a difference in overall performance.

felixwrt avatar Jul 24 '23 13:07 felixwrt

Additionally it would be helpful to explore what gets initialised in the code when the tray windows is opened for the first time after starting nextcloud client. @felixwrt As you mention in https://github.com/nextcloud/desktop/issues/4918#issuecomment-1636300569 not opening the tray at all results in higher transfer rates.

PhilippSchlesinger avatar Jul 25 '23 08:07 PhilippSchlesinger