artefactual
Problem: Loading the Archival Institution takes very long when folder is big
Current Behavior
Steps to reproduce the behavior
- Go to '...' Select Archival Description Select Archival Institution (Click picture top left) Loading takes very long. Bad Gateway I extended time-out and it works but take a very long time to display 13TB of images on mounted drive in Archival Institution I commented out "Upload limit" and it was quick in repository/actions/uploadLimitComponent.class.php
Expected Behavior
It should do the space usage and checking quicker
Possible Solution
No response
Context and Notes
No response
Version used
No response
Operating System and version
No response
Default installation culture
No response
PHP version
No response
Contact details
No response
This stems from the RecursiveDirectoryIterator logic for calculating directory size here.
This is unfortunately going to be slow when there are a lot of files, but this is one of the few OS agnostic ways of getting a directory size, and I think if we did change this logic, it would be good to not depend on system commands or utilities for this. PHP does now have disk_total_space but this would require some testing to see if that's faster.
I did some tests DISK/FOLDER SIZE CALCULATION METHODS COMPARISON Testing directory: /usr/share/nginx/atom/uploads/r/rock-art-research-institute PHP Version: 7.4.3-4ubuntu2.29 METHOD: RecursiveDirectoryIterator (Standard PHP) ============ Size: 1.21 TB (1,332,907,041,511 bytes) Execution time: 96.3082 seconds Execution time: 96,308.21 milliseconds METHOD: scandir() recursive function: Size: 1.21 TB (1,332,907,041,511 bytes) Execution time: 75.7756 seconds METHOD: System 'du' command (Linux/Unix): Size: 1.21 TB (1,332,933,107,875 bytes) Execution time: 79.1971 seconds METHOD: System 'find' + 'stat' command (Linux/Unix): Size: 1.21 TB (1,332,907,041,511 bytes) Execution time: 34.7002 seconds
METHOD: du -sb (disk usage in bytes) - FASTEST: Size: 1.21 TB (1,332,933,107,875 bytes) Execution time: 72.238890 seconds METHOD: du -s (1K blocks) then convert: Size: 1.21 TB (1,333,296,578,560 bytes) Execution time: 29.769917 seconds METHOD: Cache/Database approach (fastest for repeat queries): Size: 1.21 TB (1,332,933,107,875 bytes) Execution time: 36.667479 seconds