files:scan" the process stops scanning Files after a Thumbs.db file is hit

[IgnaceSannen]

⚠️ This issue respects the following points: ⚠️

• [X] This is a bug, not a question or a configuration/webserver/proxy issue.
• [X] This issue is not already reported on Github (I've searched it).
• [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Nextcloud version: 24.0.2 OS: Ubuntu 22.04 running in LXC container From the system info: Operating System: Linux 5.13.0-51-generic x86_64 Fresh install. This issue is happening on external storage, mounted with SMB. When running following command "sudo -u www-data /usr/bin/php /srv/nextcloud/occ -vvv files:scan" the process stops scanning files when a Thumbs.db file is hit. So the output of the above command gives something like this: Folder /user/files/mountpoint/FOLDER1 File /user/files/mountpoint/FOLDER/FILE1 File /user/files/mountpoint/FOLDER/FILE2 File /user/files/mountpoint/FOLDER/Thumbs.db Folder /user/files/mountpoint/FOLDER2 Folder /user/files/mountpoint/FOLDER3 Folder /user/files/mountpoint/FOLDER4

The outcome of this behavior is that in nextcloud FOLDER2, 3 and 4 are empty. I deleted Thumbs.db with rm and the issue was solved. Since I just wiped Thumbs.db, I did not know what was in the file. But when I create a file with the same name and just 'test' as content the same issue appears.

Steps to reproduce

1. create file Thumbs.db in a folder.

Expected behavior

Scan should run without issue, even with thumbs.db file in it.

Installation method

All components installed (no snap install)

Operating system

22.04 in LXC

PHP engine version

8.1.2

Web server

No response

Database engine version

Type: mysql Version: 10.6.7

Is this bug present after an update or on a fresh install?

No, it is a fresh install

Are you using the Nextcloud Server Encryption module?

No

What user-backends are you using?

• [ ] Default user-backend (database)
• [ ] LDAP/ Active Directory
• [ ] SSO - SAML
• [ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "myprivateip",
            "myFQDN"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.2.1",
        "overwrite.cli.url": "http:\/\/myprivateip\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "overwritehost": "myFQDN",
        "overwritewebroot": "\/nextcloud",
        "overwritecondaddr": "myprivate_ip",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - bruteforcesettings: 2.4.0
  - circles: 24.0.0
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - external: 4.0.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_external: 1.16.1
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - password_policy: 1.14.0
  - passwords: 2022.6.20
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - recommendations: 1.3.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - support: 1.7.0
  - survey_client: 1.12.0
  - systemtags: 1.14.0
  - tasks: 0.14.4
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - unsplash: 1.2.5
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - viewer: 1.8.0
  - weather_status: 1.4.0
  - workflowengine: 2.6.0
Disabled:
  - admin_audit
  - encryption
  - photos: 1.6.0
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[ruppo68]

Same problem here. We mount a smb share. In the Extern storage Config is Show hidden files not selected. There are Thumbs.db in the Filesystem, with the Explorer they are not shown, because they have the Attribute HS. But the file scan in the Background inventories the file !!!!!! These causes a download of a folder with a file Thumbs.db in it as a zip file generates a corrupt zip file. Nextcloud Release is the same.

Regards, Peter Rupp

[IgnaceSannen]

Another odd behavior, after deletion of the Thumbs.db file, the scan process runs without issues. But since we are dealing with external storage, which is also accessible through SMB, I scheduled a scan job every night. I see now that exact the same folders are empty again and the Thumbs.db file has reappeared.

[mobilandi]

Hi all,

I'm not sure if my problem is related to this one but I'll add the info just in case:

• I'm currently on nextcloud 24.0.4 but the problem exists for longer.
• When trying to access a directory on external storage (mounted via SFTP) I often get an error pop-up "Operation not allowed"
• When this happens I open the directory on the storage and delete the "thumbs.db" in it.
• After removing the thumbs.db I can access the directory via nextcloud without any issues.
• A new thumbs.db gets created after time, but accessing the directory most of the time is still possible. Sometimes it happens that I have to repeat the process for the same directory after some time.

[sierra2256]

Same issue here. files:scan crashes when it reaches Thumbs.db

Exception during scan: Invalid path: files/files/xxxxxxxxxx/xxxxxxxx/xxxxxxx/xxxxxx/Thumbs.db #0 /var/www/nextcloud/lib/private/Files/Storage/Local.php(253): OC\Files\Storage\Local->getSourcePath() #1 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(227): OC\Files\Storage\Local->file_exists() #2 /var/www/nextcloud/lib/private/Files/View.php(1345): OC\Files\Storage\Wrapper\Wrapper->file_exists() #3 /var/www/nextcloud/lib/private/Files/View.php(1390): OC\Files\View->getCacheEntry() #4 /var/www/nextcloud/lib/private/Metadata/FileEventListener.php(83): OC\Files\View->getFileInfo() #5 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(87): OC\Metadata\FileEventListener->handle() #6 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(251): OC\EventDispatcher\ServiceEventListener->__invoke() #7 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(73): Symfony\Component\EventDispatcher\EventDispatcher->callListeners() #8 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(88): Symfony\Component\EventDispatcher\EventDispatcher->dispatch() #9 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(100): OC\EventDispatcher\EventDispatcher->dispatch() #10 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(235): OC\EventDispatcher\EventDispatcher->dispatchTyped() #11 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils{closure}() #12 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array() ...

Deleting Thumbs.db from file system does not help and issue persists.

Nextcloud Version: 24.0.5.1

[sierra2256]

I found the issue - it relates to my local installation.

I had added "Thumbs.db" to blacklisted_files property in config.php. Now I removed that from blacklist and it works fine again.

[luxifr]

Soemthing similar here... it seems the scanning generally stops when it hits a file that causes any kind of exception

Exception during scan: getimagesizefromstring(): Corrupt JPEG data: 14 extraneous bytes before marker
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler(2, 'getimagesizefro...', '/var/www/html/l...', 71)
#1 /var/www/html/lib/private/Metadata/Provider/ExifProvider.php(71): getimagesizefromstring('\xFF\xD8\xFF\xE1\x02\x12Exif\x00\x00II*...')
#2 /var/www/html/lib/private/Metadata/MetadataManager.php(68): OC\Metadata\Provider\ExifProvider->execute(Object(OC\Files\Node\File))
#3 /var/www/html/apps/files/lib/Command/Scan.php(138): OC\Metadata\MetadataManager->generateMetadata(Object(OC\Files\Node\File), false)
#4 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}('/username/files/Do...')
#5 /var/www/html/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array(Object(Closure), Array)
#6 /var/www/html/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit('\\OC\\Files\\Utils...', 'scanFile', Array)
#7 /var/www/html/lib/private/Files/Utils/Scanner.php(131): OC\Hooks\PublicEmitter->emit('\\OC\\Files\\Utils...', 'scanFile', Array)
#8 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}('files/Dokumente...', 'home::username')
#9 /var/www/html/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array(Object(Closure), Array)
#10 /var/www/html/lib/private/Files/Cache/Scanner.php(170): OC\Hooks\BasicEmitter->emit('\\OC\\Files\\Cache...', 'scanFile', Array)
#11 /var/www/html/lib/private/Files/Cache/Scanner.php(445): OC\Files\Cache\Scanner->scanFile('files/Dokumente...', 3, 323042, Object(OC\Files\Cache\CacheEntry), true, Array)
#12 /var/www/html/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren('files/Dokumente...', true, 3, 323042, true, 0)
#13 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323042, true)
#14 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323037, true)
#15 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323034, true)
#16 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 322890, true)
#17 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente', true, 3, 320870, true)
#18 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 92738, true)
#19 /var/www/html/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren('', true, 3, 92734, true, Array)
#20 /var/www/html/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan('', true, 3)
#21 /var/www/html/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan('/username', true, NULL)
#22 /var/www/html/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles('username', '/username', true, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#23 /var/www/html/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /var/www/html/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /var/www/html/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/html/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /var/www/html/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#28 /var/www/html/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/html/console.php(100): OC\Console\Application->run()
#30 /var/www/html/occ(11): require_once('/var/www/html/c...')
#31 {main}

This is a bit of a pain when trying to mass-load data into nextcloud externally and then trying to scan them into the index

[szaimen]

Hi, please update to 25.0.7 or better 26.0.2 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 26-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

[joshtrichards]

Hi @IgnaceSannen Thanks for the report.

What is your SMB server? Is it Samba by chance? Check if you have any veto files lines in your smb.conf. Sometimes Thumbs.db gets added there.

@mobilandi While your report involves Thumbs.db I don't think it's being caused by the same situation. Please open a separate Issue for your situation and provide your details including logs.

@ruppo68 Hidden files have different definitions across operating systems. The System/Hidden attributes under Windows are not the same as what Unix environments consider hidden (which are files that start with dots - e.g. .mysecretfile). There are different ways of dealing with your situation. I suggest posting/searching at the Nextcloud Help Forum. Not clear about the zip issue, but that sounds interesting enough I'd suggest opening an Issue about it since it's outside the scope of this Issue.

@sierra2256 That makes sense. The telltale sign of hidden a blacklisted file is that Invalid path: line. Glad you got it worked out.

@luxifr Unrelated to this particular Issue.