Lychee
Lychee copied to clipboard
LycheeOrg
Album downloads are too taxing on the server
Each time a user tries to download an album Lychee tries to compress all the photos in it, which for big albums can be quite taxing in both RAM and CPU usage, and in many cases ends with the operation being killed without more feedback to the user. I understand compressing files is a CPU bound operation, but maybe it could be explored another option (gradually adding files to a cached zip folder when importing photos into a folder? Compressing files in small batches?), or, at the very least, provide the user with some feedback on what went wrong
Output of the diagnostics [REQUIRED]
Diagnostics
-------
Warning: Dropbox import not working. dropbox_key is empty.
Info: Latest version of PHP is 8.1
System Information
--------------
Lychee Version (release): 4.4.0
DB Version: 4.4.0
composer install: --no-dev
APP_ENV: production
APP_DEBUG: false
System: Linux
PHP Version: 8
PHP User agent: Lychee/4 (https://lycheeorg.github.io/)
Max uploaded file size: 500M
Max post size: 500M
Max execution time: 200
MySQL Version: 10.5.15-MariaDB-cll-lve
Imagick: 1
Imagick Active: 1
Imagick Version: 1808
GD Version: 2.2.5
Config Information
--------------
version: 040400
check_for_updates: 0
sorting_Photos_col: taken_at
sorting_Photos_order: ASC
sorting_Albums_col: max_taken_at
sorting_Albums_order: ASC
imagick: 1
skip_duplicates: 0
small_max_width: 0
small_max_height: 360
medium_max_width: 1920
medium_max_height: 1080
lang: es
layout: 1
image_overlay_type: desc
default_license: reserved
compression_quality: 90
full_photo: 1
delete_imported: 0
Mod_Frame: 1
Mod_Frame_refresh: 30
thumb_2x: 1
small_2x: 1
medium_2x: 1
landing_page_enable: 0
landing_owner: Manuel Genovés
landing_title: Manuel Genovés
landing_subtitle:
landing_facebook:
landing_flickr:
landing_twitter: https://www.twitter.com/Manuelgenoves
landing_instagram: https://instagram.com/escapistes
landing_youtube:
landing_background: dist/cat.jpg
site_title: Manuel Genovés
site_copyright_enable: 1
site_copyright_begin: 2019
site_copyright_end: 2021
additional_footer_text:
display_social_in_gallery: 0
public_search: 0
SL_enable: 0
SL_for_admin: 0
public_recent: 0
recent_age: 1
public_starred: 0
downloadable: 1
photos_wraparound: 1
map_display: 1
zip64: 1
map_display_public: 0
map_provider: Wikimedia
force_32bit_ids: 1
map_include_subalbums: 0
update_check_every_days: 3
has_exiftool: 0
share_button_visible: 0
import_via_symlink: 0
has_ffmpeg: 0
location_decoding: 0
location_decoding_timeout: 30
location_show: 1
location_show_public: 0
rss_enable: 0
rss_recent_days: 7
rss_max_items: 100
prefer_available_xmp_metadata: 0
editor_enabled: 1
lossless_optimization: 0
swipe_tolerance_x: 150
swipe_tolerance_y: 250
local_takestamp_video_formats: .avi|.mov
log_max_num_line: 1000
unlock_password_photos_with_url_param: 0
nsfw_visible: 1
nsfw_blur: 0
nsfw_warning: 0
nsfw_warning_admin: 0
map_display_direction: 1
album_subtitle_type: oldstyle
upload_processing_limit: 4
public_photos_hidden: 1
new_photos_notification: 0
How large an album are we talking about? Your Max execution time is 200 (CPU seconds) which should be enough for quite a few I would think?
Now, a point can be made that photo/video media files are compressed already and having Lychee try to compress them again is a waste of time anyway. I'll see if there is an option in the Zip compressors we use to disable compression...
or, at the very least, provide the user with some feedback on what went wrong
This will be solved by a big, pending re-factoring which largely improves on error handling. Note, that the user won't see a direct error message, because the ZIP archive is downloaded via an ordinary HTTP GET request. But at least the logs will contain a proper error message.
gradually adding files to a cached zip folder
Actually, the download should already start streaming to the client while more photos are added to the ZIP archive progressively.
I'm talking about an album with 78 photos, 7-10mb each one. My hosting provides 1,5GB of RAM which get exhausted. CPU gets to a 100%, but I'm not sure what the specs are in that regard
I wouldn't expect RAM to be an issue because, as @nagmat84 wrote, the zip file is streamed to the client while media files are added to it, so the total archive size shouldn't matter for the server. Our Zip compressor is supposed to use twice as much RAM as the current source file being compressed, so it shouldn't be more than some 20 MB in your case.
Let's try to disable the compression though and see if that helps. Our Zip compressor already does that for files that it considers "large", but the default threshold is at 20 MB. You are on the 4.4.0 release, correct? Can you modify the following file: https://github.com/LycheeOrg/Lychee/blob/5b51765c7c66e6542ff2a6dcd3fb57393ff16717/app/Actions/Album/Archive.php#L40-L42
Add the following line between the current lines 41 and 42:
options->setLargeFileSize(0);
Can you then try to download the album again and let us know if it works better now? Thanks!
Now it hangs for a while and then the page "unloads" to say so.
See this example: *****
This sounds a little like a buffering problem. How does your setup look like? Which HTTP server and how does PHP interact with the server (included, cgi, php-fpm)?
If the webserver tries to buffer the output of Lychee and waits until the whole process has been terminated, this would also explain the memory consumption.
Agreed. I tried with smaller albums, like "EMOZ" and "Zaragoza", and I was able to download those, but at least on the first attempt there was a curious delay before the download started (like 5-6 seconds long), for which there really is no explanation. It seems like something is caching the responses on the server side. Are you running some sort of a proxy?
Not that I know. This is a shared hostinger (hosting24) server, with some interesting perks (unlimited storage and bandwidth). I don't know exactly how they run the servers, but I have a fair amount of access to it (even ssh) and I couldn't see a proxy or anything like that
I do not think this is a problem with Lychee but with a config of the server, we are using ZipStream especially in order to avoid such issue (before we were building the zip before sending it).
https://photography.viguier.nl/gallery#JfdRU2VvM5MgZbfPNiocgAdq You can download this album of 573 pictures and 1.2GB (1.7 uncompressed). The RAM usage on my server is at 390MB over 1.9GB free when downloading and gets down to 363MB when idle. So I have strong believe this is NOT a Lychee problem.
@somas95 It has not necessarily to be a dedicated proxy. I myself have recently run into a similar problem with Apache in combination with PHP-FPM and a particular set of Apache extensions. Apache as well as Nginx have quite a fair amount of caching capabilities built into them. So it would be really interesting to know your server configuration.
yeah, I do have php-fpm. The list of modules is this one, not sure if you would need something more (I'm cautious about posting a phpinfo dump in here):
PHP Modules
- bcmath
- bz2
- calendar
- clos_ssa
- Core
- ctype
- curl
- date
- dom
- exif
- fileinfo
- filter
- ftp
- gd
- gettext
- gmp
- hash
- i360
- iconv
- imagick
- imap
- json
- libxml
- mbstring
- mysqli
- mysqlnd
- openssl
- pcntl
- pcre
- PDO
- pdo_mysql
- pdo_sqlite
- Phar
- posix
- pspell
- readline
- Reflection
- session
- shmop
- SimpleXML
- soap
- sockets
- SPL
- standard
- tidy
- tokenizer
- xml
- xmlreader
- xmlwriter
- xsl
- zip
- zlib
I meant your Apache modules. Please run apache2ctl -M and update your post.
What do you mean? I thought you have SSH access?
apache2ctl is a shell script, maybe your hoster doesn't use that. In case of the option -M, the shell script is only a thin wrapper around the Apache executable anyway. Running apache2 -M should give the exact same result.
And I believe that Debian renames the Apache executable to httpd. In that case the correct command would be httpd -M, but I am not sure about that.
I thought I had, but seems I only have access to a limited jailed sh session. I already tried httpd and apache2ctl before, just tried apache2 now and it's the same. So I guess I don't know which modules do they load
Never mind. The fact that you are using Apache together with PHP-FPM suggests that you might suffer from the same configuration error which I myself had encountered recently. The issue only occurs in combination with the Apache modules mod_gzip or mod_deflate. That's why I was curious to know what Apache modules are loaded.
You need to disable the compression functionality which is provided by these two modules for certain URLs otherwise you will end up with a "double" compression. The fix is part of PR #1217. However, you can apply the needed changes locally to your .htacess in the public/ folder by hand. The new .htaccess can be found here
https://github.com/LycheeOrg/Lychee/blob/4e22447388afdc6d9857a410682422ee305c55c5/public/.htaccess
The crucial part are the RewriteRule which set no-gzip=1 for certain URLs. This disables the modules mod_gzip or mod_deflate for these URLs.
Let me know, if this fixes the problem for you.
BTW: The potential bug fix has been merged into the master branch. Instead of trying to fix .htaccess manually, you can also update to the head of master, if this is a viable option for you.
I tried editing the .htaccess with no avail. Reviewing usage stats seems that I consistently hit an I/O usage cap of 3mb/s, but that should only throttle the process, no kill it. So I'm still in the dark, albeit I guess I can't expect much from a shared server