android icon indicating copy to clipboard operation
android copied to clipboard

Published 20 hours ago • verified publisher icon nextcloud

Auto Upload Fails, InstantUpload folder navigation "Server not available"

Open btwood opened this issue 1 year ago • 2 comments

⚠️ Before posting ⚠️

  • [X] This is a bug, not a question or an enhancement.
  • [X] I've searched for similar issues and didn't find a duplicate.
  • [X] I've written a clear and descriptive title for this issue, not just "Bug" or "Crash".
  • [X] I agree to follow Nextcloud's Code of Conduct.

Steps to reproduce

Auto Upload:

  1. Open Nextcloud App
  2. Go to "Settings" -> "Auto Upload"
  3. Disable and Reenable Camera folder upload (try to force sync)

Folder Navigation:

  1. Select "All Files"
  2. "InstantUpload"
  3. "Camera"
  4. Year folder
  5. Month folder
  6. Thumbnails don't load, "Server not available" is displayed.

Expected behaviour

Auto Upload will queue and send files to the server. Ideally at least once a day. Folder navigation renders thumbnails.

Actual behaviour

Auto Upload doesn't seem to detect any new files. The "Upload" page is empty. The last folder I have is from last month (2024/01), and the last image is from the 31st. I'm not sure if there is an issue with YYYY/MM folder structure, or something else. Selecting an image to view, seems to actually displays it, so there isn't an issue with retrieving the file.

Nextcloud App has full permissions, and battery optimization is disabled.

Server side I get errors about generating metadata for files. I spot checked one of them, and it's a JPEG from 2018 that previously had no issues. That doesn't seem to be directly related to the upload issue.

Web app on a computer navigates and renders thumbnails faithfully. No issues with using the web-page directly.

Android version

14

Device brand and model

Samsung Galaxy S21+ 5G

Stock or custom OS?

Stock

Nextcloud android app version

3.27.0

Nextcloud server version

28.0.2

Using a reverse proxy?

I don't know

Android logs

logcatOutput.txt

Server error logs

{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T06:41:48+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 459079","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"OCP\\FilesMetadata\\Exceptions\\FilesMetadataException","Message":"json cannot exceed 100000 characters long","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":116,"function":"saveMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","Line":179,"message":"Error while generating metadata for fileid 459079","exception":{},"CustomMessage":"Error while generating metadata for fileid 459079"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T06:42:45+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 457694","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"OCP\\FilesMetadata\\Exceptions\\FilesMetadataException","Message":"json cannot exceed 100000 characters long","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":116,"function":"saveMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","Line":179,"message":"Error while generating metadata for fileid 457694","exception":{},"CustomMessage":"Error while generating metadata for fileid 457694"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T06:43:12+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 458576","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"OCP\\FilesMetadata\\Exceptions\\FilesMetadataException","Message":"json cannot exceed 100000 characters long","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":116,"function":"saveMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","Line":179,"message":"Error while generating metadata for fileid 458576","exception":{},"CustomMessage":"Error while generating metadata for fileid 458576"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T06:44:54+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 465401","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"OCP\\FilesMetadata\\Exceptions\\FilesMetadataException","Message":"json cannot exceed 100000 characters long","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":116,"function":"saveMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","Line":179,"message":"Error while generating metadata for fileid 465401","exception":{},"CustomMessage":"Error while generating metadata for fileid 465401"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":3,"time":"2024-02-18T07:42:23+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"$absolutePath must be relative to \"files\", value is \"/\"","userAgent":"--","version":"28.0.2.5","data":[]}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T07:42:23+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 14277","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"InvalidArgumentException","Message":"$absolutePath must be relative to \"files\"","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1944,"function":"getPathRelativeToFiles","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":2052,"function":"lockPath","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1141,"function":"lockFile","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":987,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/Node/File.php","line":116,"function":"fopen","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php","line":71,"function":"fopen","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":86,"function":"handle","class":"OCA\\Photos\\Listener\\ExifMetadataProvider","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":230,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":59,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":94,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":106,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":115,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/Files/View.php","Line":2126,"message":"Error while generating metadata for fileid 14277","exception":{},"CustomMessage":"Error while generating metadata for fileid 14277"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":3,"time":"2024-02-18T07:43:53+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"$absolutePath must be relative to \"files\", value is \"/\"","userAgent":"--","version":"28.0.2.5","data":[]}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T07:43:53+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 13478","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"InvalidArgumentException","Message":"$absolutePath must be relative to \"files\"","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1944,"function":"getPathRelativeToFiles","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":2052,"function":"lockPath","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1141,"function":"lockFile","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":987,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/Node/File.php","line":116,"function":"fopen","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php","line":71,"function":"fopen","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":86,"function":"handle","class":"OCA\\Photos\\Listener\\ExifMetadataProvider","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":230,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":59,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":94,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":106,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":115,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/Files/View.php","Line":2126,"message":"Error while generating metadata for fileid 13478","exception":{},"CustomMessage":"Error while generating metadata for fileid 13478"}}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":3,"time":"2024-02-18T07:50:25+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"$absolutePath must be relative to \"files\", value is \"/\"","userAgent":"--","version":"28.0.2.5","data":[]}
{"reqId":"cALPGxqeZAQDraZZ83VI","level":2,"time":"2024-02-18T07:50:25+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"Error while generating metadata for fileid 1446","userAgent":"--","version":"28.0.2.5","exception":{"Exception":"InvalidArgumentException","Message":"$absolutePath must be relative to \"files\"","Code":0,"Trace":[{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1944,"function":"getPathRelativeToFiles","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":2052,"function":"lockPath","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":1141,"function":"lockFile","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/View.php","line":987,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/lib/private/Files/Node/File.php","line":116,"function":"fopen","class":"OC\\Files\\View","type":"->"},{"file":"/data/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php","line":71,"function":"fopen","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":86,"function":"handle","class":"OCA\\Photos\\Listener\\ExifMetadataProvider","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":230,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/data/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":59,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":94,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":106,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":115,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":109,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":99,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":88,"function":"scanFolder","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php","line":74,"function":"scanFilesForUser","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/Job.php","line":81,"function":"run","class":"OC\\Core\\BackgroundJobs\\GenerateMetadataJob","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":102,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/data/www/nextcloud/lib/public/BackgroundJob/TimedJob.php","line":92,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->"},{"file":"/data/www/nextcloud/cron.php","line":152,"function":"execute","class":"OCP\\BackgroundJob\\TimedJob","type":"->"}],"File":"/data/www/nextcloud/lib/private/Files/View.php","Line":2126,"message":"Error while generating metadata for fileid 1446","exception":{},"CustomMessage":"Error while generating metadata for fileid 1446"}}

Additional information

No response

btwood avatar Feb 18 '24 21:02 btwood

I just wanted to add some info, and maybe some speculation about what is going on.

I'm not a "power user", but I do take a lot of photos. My camera folder / gallery has 20,000+ images in it. It seems like the nextcoud application is very inefficient at reading files and deciding what to do with them. Another user on my server has a screenshots folder in the 10,000's and has a similar issue.

Not knowing anything about the underlying application. Perhaps having a local sqlite database or cache file or something with all the files and hashes and some time metadata, and then do bulk comparison against the server would help? I don't know what the current strategy is, but if it's going file by file trying to compare hashes against the server, that would cause a lot of issues. Many-small-files problems. Etc. The typical solutions for dealing with tons of small files is to bulk process, group, and reduce the workload as much as possible. Cache the info you need, divide the work, bundle it into "chunks" of 100 to 1000, etc.

I don't know how the native android gallery or photos reads images, but it seems much more performant than the nextcloud app.

I noticed issues starting around 3,000 photos, but resolved most of it by moving to year/month folder structure server side.

I recently attempted to manually add the files by copying everything off of my phone using 'adb-sync' and manually adding it to nextcloud. I wrote a short bash script to chunk things out into yyyy/mm folders as well. It didn't seem to work. Nextcloud is seemingly insisting on checking every file.

btwood avatar May 01 '24 01:05 btwood

I've thought some more about this. You don't really need a database for object culling. You could simply leverage file metadata and timestamps. Preserve/save the last modified time of the last successful upload, and you can prune everything prior to that.

This would make the task of locating and culling files as easy as traversing the filesystem. Using adb shell the find -mtime command is very performant. I can get a complete listing of a full phone (~240GB used) within a few minutes. If I add -mtime -1 to look at stuff from the past day, it returns in seconds.

Perhaps there are some inefficiencies with how the app finds and culls files?

I did recently discover that HTML/2 is a large improvement over HTML/1.1. It is too soon to tell, but I've enabled it on my instance in hopes the auto upload is better. a2enmod http2

Just more random thoughts. I hope someone is able to look at this and improve how the app handles many files and nearly full devices.

btwood avatar Aug 25 '24 18:08 btwood