fileshelter
fileshelter copied to clipboard
problem with uploading a huge number of files
I have over 10,000 .blob files, I added them to a .7z (tried to upload before .rar, .zip) archive which ended up being about 40gb in size. Then I tried to upload the archive to the fileshelter, but the process stopped at 100%, I waited all weekend just in case. Further, I thought that the system was trying to get inside the archive, but after making the archive with a password I got the same result. For the sake of interest, I tried to upload .iso file and .bak database backup (each file is 40 GB in size) - it uploaded almost instantly without any problems. At the same time, there are zero errors in the fileshelter.log. Can you help solve the problem?
Well this is strange. Fileshelter does not try to open your archive. So uploading a .7z file or a .iso file does not matter and should be the same.
You can try to run with full logs by tweaking this:
https://github.com/epoupon/fileshelter/blob/a4a3b23f1bd4b9cc14b0839d66541ddfc0369099/conf/fileshelter.conf#L10
Change to log-config = "*";
corrected the config as you said, in the screenshot the last entries in the log when the upload progress was stuck at 100%
@epoupon if you are interested, I can give such an archive by uploading it, for example, to Google Drive
I am having trouble uploading files as well. I just set it up in docker, and when I upload files, sometimes they just work, other times it just gets stuck at 0% upload:
It gets stuck at 0% in about 80-90% of cases larger than about 1 megabyte, I'd say. There is no helpful entry in the logs at all for this, even with the modification you suggested above. The only thing that appears is [info] "WebRequest: took 0.81 ms". I'm not really sure how to proceed. I run fileshelter via docker and behind nginx-proxy. I have tried setting behind-reverse-proxy to true and false without any noticeable difference.
This is my docker-compose:
services:
fileshelter:
image: epoupon/fileshelter
container_name: fileshelter
user: "1000:1000"
network_mode: bridge
restart: always
# ports:
# - "5090:5091" #WebUI
environment:
- VIRTUAL_PORT=5091
- VIRTUAL_HOST=fileshelter.my.domain.tld
- LETSENCRYPT_HOST=fileshelter.my.domain.tld
volumes:
- "/home/bsc/fileshelter/var:/var/fileshelter:rw"
- "/home/bsc/fileshelter/fileshelter.conf:/etc/fileshelter.conf:rw"
and here is my fileshelter.conf: fileshelter.txt
Edit: This was my fault. I had the client_max_body_size set to the default of 1m. Info on how to change that is available at nginx-proxy.