nextcloud_ynh
nextcloud_ynh copied to clipboard
YunoHost-Apps
Transfer very slow with large among of files
Hi all,
When I try to transfer a large among of (small) files (around 65000), it takes a huge time, more than 7 days (estimated by nextcloud client). It takes something than 1 or 2 seconds per file. I already see this Nextcloud optimizations, but I'm not sure that it will not break my nextcloud instance on the next update. And it's not clear to me what to change exactely. Is there other people who has this issue? How did you solve this? Is it plan to fix this in the next upgrade of the application nextcloud in Yunohost?
Hi,
Did you tried to tar your files, upload the tar, and untar it? (I'm not sure it will works)
Hi,
Did you tried to tar your files, upload the tar, and untar it? (I'm not sure it will works)
I don't think that I can do that. Nrextcloud instance is not able to uncompressed a file on his stotage. And in all the case, the problem that I want to resolve is reducing the time between 2 files upload. But thank you.
You can with this app: https://apps.nextcloud.com/apps/extract
Good to know this app. But, it's not what I want. I want to synchronized all my changes. I don't want to upload a compress file every time I change something.
Good to know this app. But, it's not what I want.
Yeah I understood, I thought it was still useful to mention it :)
Did you try the FlowUpload app ? Does it work correctly ? (I'm asking to know where the issue can be on your Nextcloud)
Good to know this app. But, it's not what I want.
Yeah I understood, I thought it was still useful to mention it :)
Did you try the FlowUpload app ? Does it work correctly ? (I'm asking to know where the issue can be on your Nextcloud)
No, I didn't try. I will try it this evening and I will keep you informed.
Good to know this app. But, it's not what I want.
Yeah I understood, I thought it was still useful to mention it :)
Did you try the FlowUpload app ? Does it work correctly ? (I'm asking to know where the issue can be on your Nextcloud)
Hi,
sorry to give you answer so late, I didn't found time to test before.
So, I tested FlowUpload, and it's working well with many files. After that, I decided to test by uploading a big file (700 Mo) directly via the web page, and it takes 5 or 6 minutes. I tested again with many files, via the web page, and I had an sql error: Doctrine\DBAL\Exception\DriverException: An exception occurred while executing 'SELECT fileid, storage, path, path_hash, parent, name, mimetype, mimepart, size, mtime, storage_mtime, encrypted, etag, permissions, checksumFROMoc_filecacheWHEREstorage= ? ANDpath_hash = ?' with params [2, "58de2ce24cd0e10c8be8d076db93f0e7"]: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away
I didn't checked yet on the net what it means, but maybe it's a way to investigate. Is someone has already see that?
Thank you.
I don't know if this is the cause of your problem, but quite a while back I found that when I tried to sync a large number of files that Nextcloud was essentially using all of the RAM, then getting heavily into using the swap instead and then when that ran out crashing php.
I never found a fix, but I would guess that any fix is likely to mainly be in the Nextcloud codebase so it throttles itself based on the RAM available, thus avoiding using the swap.
