owncloud
Library scan fails on SFTP mounted storage
Behaviour:
- Music app offers to scan new entries
- Scan is very slow, processes 10 at a time
- Stops after a short while with progress unchanged until page refresh
Debug logs:
Error PHP Undefined array key "mpeg" at 2024-07-20T19:27:27+00:00
/srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731
Debug music Scanning 10 files of user 2024-07-20T19:27:30+00:00
me
Debug music Scanning 10 files of user 2024-07-20T19:27:34+00:00
me
Debug music Scanning 10 files of user 2024-07-20T19:27:38+00:00
me
Debug music Scanning 10 files of user 2024-07-20T19:27:42+00:00
me
Debug music Scanning 10 files of user 2024-07-20T19:27:46+00:00
me
Debug no app in context Logging out 2024-07-20T19:27:48+00:00
Debug no app in context Session token invalidated before logout 2024-07-20T19:27:48+00:00
Error PHP Trying to access array offset on value 2024-07-20T19:27:48+00:00
of type null at
/srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731
Error PHP Trying to access array offset on value 2024-07-20T19:27:48+00:00
of type null at
/srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731
Error PHP Undefined array key "mpeg" at 2024-07-20T19:27:48+00:00
/srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731
Error index OCP\Files\NotFoundException: 2024-07-20T19:27:48+00:00
/me/files/n at
lib/private/Files/Node/Root.php line
207
0. .../LazyUserFolder.php line 72
OC\Files\Node\Root->get(
)
1. .../Util.php line 332
OC\Files\Node\LazyUserFolder->get(
)
2. .../LibrarySettings.php line 107
OCA\Music\Utility\Util::getFolderFromRelativePath(
)
3. .../Scanner.php line 514
OCA\Music\Utility\LibrarySettings->getFolder(
)
4. .../ApiController.php line 213
OCA\Music\Utility\Scanner->scanFiles(
)
5. .../Dispatcher.php line 230
OCA\Music\Controller\ApiController->scan(
)
6. .../Dispatcher.php line 137
OC\AppFramework\Http\Dispatcher->executeController(
)
7. .../App.php line 184
OC\AppFramework\Http\Dispatcher->dispatch(
)
8. .../Router.php line 315
OC\AppFramework\App::main(
)
9. .../base.php line 1069
OC\Route\Router->match(
)
10. index.php line 39
OC::handleRequest(
)
Debug music Scanning 10 files of user 2024-07-20T19:27:48+00:00
me
Thanks for the report.
According the timestamps in your log, scanning each 10 files patch has taken about 4 seconds, i.e. 0.4 seconds per file. I don't think this is unreasonable, considering that each file has to be opened and parsed individually and they reside on another server. I'm not familiar with the SFTP external storages, but depending on its implementation, parsing the contents of the files on the storage may require downloading the entire file from the remote server to your OC/NC server. Or if the integration is smart enough, then partial download may be enough. Using internal storage, you should see 5-10 times faster scan speed.
However, the scanning stopping entirely after a while is unexpected. When the scanning has stopped and you reload the page, does the scanning then again work for a short while before stopping again, or do you need to erase the previously scanned files before you can scan again?
Curiously, the log looks like the user had been logged out in the middle of the scanning process and then reading the files started failing (probably because the access to the storage was lost due to the log-out).
One thing to try would be to do the scanning from the command line, and see, if it makes any difference. The command to give in the main folder of owncloud or nextcloud is something like sudo -u www-data php occ music:scan your_user_name.
If everything else fails, then you may also disable the metadata scanning from the Settings view of the Music app. You will miss out on many of the features of the Music app but at least the scanning should be much faster as the contents of the files are not read.
Well alright, I'd call it very slow.
After the refresh, it would scan a few more files and then make no more progress. I've tried the same on the command line and got no further.
Did you test without the metadata scanning? I assume that it would be a lot faster but maybe still wouldn't fix the problem of the scan stopping after a while.
BTW, what version of Nextcloud or ownCloud are you using?
BTW, what version of Nextcloud or ownCloud are you using?
Last version of 27.X and then NC 28.0.8.1
Haven't tested without metadata scanning yet; suspect there's something foul with the SFTP storage option on NC. I've just rsync'd my collection and the scan went over it quick enough.