[Bug]: Intermittend 404/200 external storage (smb/sshfs/local)

[barrydegraaff]

trafficstars

⚠️ This issue respects the following points: ⚠️

• [X] This is a bug, not a question or a configuration/webserver/proxy issue.
• [X] This issue is not already reported on Github (I've searched it).
• [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• [X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

After upgrading to Nextcloud 24.0.7 I am experiencing glitches in the web-ui on all folders that are external storage mounts. I upgraded to Nextcloud 25.0.1 but the issue remains. The issue is there for both SMB and SSH mounts, and these point to different servers on different networks.

What happens is that PROPFIND to external storage mounts seem to work one time and not the other time, I can see this in the log for example:

travel.barrydegraaff.nl - - [10/Nov/2022:19:46:26 +0100] "PROPFIND /nextcloud/remote.php/dav/files/[email protected]/Muziek/James%20Bond%20Classic%20Soundtracks HTTP/1.1" 207 1376 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"

travel.barrydegraaff.nl - - [10/Nov/2022:19:46:30 +0100] "PROPFIND /nextcloud/remote.php/dav/files/[email protected]/Muziek/James%20Bond%20Classic%20Soundtracks HTTP/1.1" 404 236 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"

As you can see the same request, sometimes works and sometimes fails.

As a result the WebUI falls back to the root, which is annoying as I am regularly +5 folders deep into a folder tree. Reloading the UI seems to always fix the issue, back-button-reload-try-again also works.

The issue occurs not only when browsing, but also download, upload and preview are affected.

Initially I thought maybe a timeout or database issue, after checking the DB settings and such, my Nextcloud performance improved, but this issue remains. Normal folders in the Nextcloud storage work without issue.

Steps to reproduce

1.Browse through external storage 2.Click on average 3-5 folders 3.Falls back to root

Expected behavior

Browse normally

Installation method

Community Manual installation with Archive

Operating system

Other

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

Updated from a minor version (ex. 22.2.3 to 22.2.4)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• [ ] Default user-backend (database)
• [ ] LDAP/ Active Directory
• [X] SSO - SAML
• [ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.*********.nl",
            "saml.**********.nl"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "25.0.1.1",
        "overwrite.cli.url": "https:\/\/nextcloud.********.nl\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG"
        ],
        "preview_max_x": 1000,
        "preview_max_y": 1000,
        "preview_max_scale_factor": 1.5,
        "jpeg_quality": 90,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "maintenance": false,
        "loglevel": 2,
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "theme": "",
        "debug": false,
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "updatechecker": true,
        "has_internet_connection": true,
        "appstoreenabled": true,
        "versions_retention_obligation": "auto, 7",
        "trashbin_retention_obligation": "auto, 7",
        "filelocking.enabled": false,
        "default_phone_region": "NL",
        "check_for_working_wellknown_setup": false,
        "skeletondirectory": "",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "profile.enabled": false
    }
}

List of activated Apps

Enabled:
  - bruteforcesettings: 2.5.0
  - calendar: 4.1.0
  - cloud_federation_api: 1.8.0
  - contacts: 5.0.1
  - dav: 1.24.0
  - external: 5.0.0
  - federatedfilesharing: 1.15.0
  - files: 1.20.1
  - files_external: 1.17.0
  - files_pdfviewer: 2.6.0
  - files_rightclick: 1.4.0
  - files_sharing: 1.17.0
  - files_trashbin: 1.15.0
  - files_versions: 1.18.0
  - logreader: 2.10.0
  - lookup_server_connector: 1.13.0
  - oauth2: 1.13.0
  - passwords: 2022.11.20
  - provisioning_api: 1.15.0
  - related_resources: 1.0.3
  - serverinfo: 1.15.0
  - settings: 1.7.0
  - text: 3.6.0
  - theming: 2.0.1
  - theming_customcss: 1.12.0
  - twofactor_backupcodes: 1.14.0
  - twofactor_totp: 7.0.0
  - user_saml: 5.0.3
  - viewer: 1.9.0
  - workflowengine: 2.7.0
Disabled:
  - activity: 2.5.2
  - admin_audit
  - circles: 22.1.1
  - comments: 1.11.0
  - contactsinteraction: 1.1.0
  - dashboard: 7.0.0
  - encryption
  - federation: 1.10.1
  - firstrunwizard: 2.1
  - nextcloud_announcements: 1.9.0
  - notifications: 2.1.2
  - password_policy: 1.3.0
  - photos: 1.0.0
  - privacy: 1.0.0
  - recommendations: 1.0.0
  - sharebymail: 1.10.0
  - support: 1.0.0
  - survey_client: 1.0.0
  - suspicious_login
  - systemtags: 1.2.0
  - updatenotification: 1.12.0
  - user_ldap: 1.15.0
  - user_status: 1.0.0
  - weather_status: 1.0.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

https://barrydegraaff.nl/share/nextcloud.log

Additional info

Fedora 36 Apache 2.4.54 PHP 8.0.25 MariaDB 10.5.16

"filelocking.enabled": false,

mysql-slow.log shows bunch of queries that are slow all seem to be relating to oc_filecache ~500000 rows

Seems the server thinks I am out of disk space, but I am not even close to being out of space. Also this error is not always thrown when the issue occurs.

{"reqId":"Y3Z2putrJroxZ61S9rRpOwAAjgE","level":3,"time":"2022-11-17T18:00:06+00:00","remoteAddr":"192.168.1.166","user":"info@****removed****.tk","app":"PHP","method":"GET","url":"/nextcloud/index.php/apps/files/ajax/getstoragestats?dir=%2FMuziek%2FJames%20Bond%20Classic%20Soundtracks","message":"disk_free_space(): No such file or directory at /var/www/html/nextcloud/lib/private/Files/Storage/Local.php#420","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36","version":"25.0.1.1","exception":{"Exception":"Error","Message":"disk_free_space(): No such file or directory at /var/www/html/nextcloud/lib/private/Files/Storage/Local.php#420","Code":0,"Trace":[{"function":"onAll","class":"OC\\Log\\ErrorHandler","type":"::","args":[2,"disk_free_space(): No such file or directory","/var/www/html/nextcloud/lib/private/Files/Storage/Local.php",420]},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Local.php","line":420,"function":"disk_free_space","args":["/var/www/html/nextcloud/data/info@****removed****.tk/files/Muziek"]},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Quota.php","line":108,"function":"free_space","class":"OC\\Files\\Storage\\Local","type":"->","args":["files/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":334,"function":"free_space","class":"OC\\Files\\Storage\\Wrapper\\Quota","type":"->","args":["files/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":1181,"function":"free_space","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["files/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/Files/View.php","line":1127,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["free_space","/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/Files/Filesystem.php","line":653,"function":"free_space","class":"OC\\Files\\View","type":"->","args":["/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/legacy/OC_Helper.php","line":407,"function":"free_space","class":"OC\\Files\\Filesystem","type":"::","args":["/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/legacy/OC_Helper.php","line":395,"function":"freeSpace","class":"OC_Helper","type":"::","args":["/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/public/Util.php","line":503,"function":"maxUploadFilesize","class":"OC_Helper","type":"::","args":["/Muziek/James Bond Classic Soundtracks",-2]},{"file":"/var/www/html/nextcloud/apps/files/lib/Helper.php","line":52,"function":"maxUploadFilesize","class":"OCP\\Util","type":"::","args":["/Muziek/James Bond Classic Soundtracks",-2]},{"file":"/var/www/html/nextcloud/apps/files/lib/Controller/AjaxController.php","line":46,"function":"buildFileStorageStatistics","class":"OCA\\Files\\Helper","type":"::","args":["/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":225,"function":"getStorageStats","class":"OCA\\Files\\Controller\\AjaxController","type":"->","args":["/Muziek/James Bond Classic Soundtracks"]},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":133,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Files\\Controller\\AjaxController"},"getStorageStats"]},{"file":"/var/www/html/nextcloud/lib/private/AppFramework/App.php","line":172,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\Files\\Controller\\AjaxController"},"getStorageStats"]},{"file":"/var/www/html/nextcloud/lib/private/Route/Router.php","line":298,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OCA\\Files\\Controller\\AjaxController","getStorageStats",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},["files.ajax.getStorageStats"]]},{"file":"/var/www/html/nextcloud/lib/base.php","line":1047,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/files/ajax/getstoragestats"]},{"file":"/var/www/html/nextcloud/index.php","line":36,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/html/nextcloud/lib/private/Log/ErrorHandler.php","Line":99,"CustomMessage":"--"}}

[barrydegraaff]

I since tried I changed some of my external storage to Local, which as these smb shares are on the same server, but even that does not mitigate the problem.

[barrydegraaff]

Tried to work around using federated sharing, while that may work it takes 30 seconds each time I open the share, so that is not a viable alternative. Disappointing.

[szaimen]

Hi, please update to 25.0.7 or better 26.0.2 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 26-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

[joshtrichards]

Does the folder /var/www/html/nextcloud/data/info@****removed****.tk/files/Muziek actually exist on your local filesystem?

The log entry is just NC trying to check your free disk space statistics, but it's failing on that path for some reason.

I noticed you're neither using Redis nor filelocking. Any particular reason?

Any memory or CPU constraints?

Might also be worth a occ files:scan

[nextcloud-command]

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

[barrydegraaff]

The issue is still present on NC 26.0.4.

/var/www/html/nextcloud/data/info@removed.tk/files/Muziek does not exist on the machine where Nextcloud is running.

PHP Version: 8.2.8 Memory limit: 828 MB Max execution time: 172800 Upload max size: 160 GB

There is enough CPU

occ files:scan does not make a difference

I stopped using external storage 6 months ago since it was unworkable. Perhaps if no-one else is complaining it is an installation specific issue.