opencloud icon indicating copy to clipboard operation
opencloud copied to clipboard
Published 3 weeks ago • verified publisher icon opencloud-eu

watchfs: possible metadata coruption after upload

Open rhafer opened this issue 1 month ago • 2 comments

Describe the bug

With STORAGE_USERS_POSIX_WATCH_FS=true, when a user changes a file on-disk while a file with the same name is upload via WebDAV and still being stuck in post-processing the attributes of the file can get corrupted.

The easiest way to reproduce this is to run the features/collaborativePosix/collaborativePosixFS.feature:55 with a prolonged POSTPROCESSING_DELAY.

Steps to reproduce

  1. Start the server with ocwrapper and an increase postprocessing delay: POSTPROCESSING_DELAY=10s tests/ocwrapper/bin/ocwrapper serve --bin=/work/opencloud/opencloud/bin/opencloud
  2. run the above mentioned test: TEST_SERVER_URL="https://localhost:9200" BEHAT_FEATURE=tests/acceptance/features/collaborativePosix/collaborativePosixFS.feature:55 make test-acceptance-api
  3. Test fails

The test upload a test file with the content contentnew via webdav, then tries to append the string new via normal via system filesystem operations. And finally tries to verify the updated content again via webdav. The content returned via webdav is now con (❗). Looking into the file on disk shows that the content is actually content (the original uploaded content), but the metadata of the file has the wrong size:

xattr -l .....
...
user.oc.blobid: f524fa38-1c8a-4eab-800f-a7e62bc7ffd3
user.oc.mtime: 2025-11-03T11:30:43.300589087Z
user.oc.parentid: 9d8f3ec4-a587-41c9-a179-8e09fdf44f89
user.oc.blobsize: 3
user.oc.id: 784bca5f-6d9a-4bb8-92ce-5dda1c2cee0c
user.oc.type: 1
user.oc.name: test.txt

The problem is also reproducible when running the above steps manually. (The manual update on disk needs to happen before the POSTPROCESSING finished)

Expected behavior

The file metadata should match the file content, whether the expected content is content or contentnew depends on whether we change happens before or during postprocessing. I'd argue that if the change happens while the file is still being processed it is ok to have the content reverted to content as long as the metadata stays consistent.

Actual behavior

Metadata and actually file disagree about the length.

test output:

Feature: create a resources using collaborative posixfs

  Background:                                                              # /work/opencloud/tests/acceptance/features/collaborativePosix/collaborativePosixFS.feature:4
    Given the config "STORAGE_USERS_POSIX_WATCH_FS" has been set to "true" # OcConfigContext::theConfigHasBeenSetTo()
    And user "Alice" has been created with default attributes              # FeatureContext::userHasBeenCreatedWithDefaultAttributes()
    And user "Alice" has created folder "/firstFolder"                     # FeatureContext::userHasCreatedFolder()

  Scenario: edit file                                                                                                  # /work/opencloud/tests/acceptance/features/collaborativePosix/collaborativePosixFS.feature:55
    Given user "Alice" has uploaded file with content "content" to "test.txt"                                          # FeatureContext::userHasUploadedAFileWithContentTo()
    When the administrator puts the content "new" into the file "test.txt" in the POSIX storage folder of user "Alice" # CliContext::theAdministratorChangesFileContent()
    Then the content of file "/test.txt" for user "Alice" should be "contentnew"                                       # FeatureContext::contentOfFileForUserShouldBe()
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:10 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000032
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:11 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000034
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:12 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000036
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:13 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000038
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:14 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000040
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:15 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000042
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:16 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000044
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ ### RESPONSE
      │ Status: 425
      │ Headers:
      │ Content-Length: 0
      │ Content-Security-Policy: child-src 'self'; connect-src 'self' blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; default-src 'none'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' blob: https://embed.diagrams.net/; img-src 'self' data: blob: https://raw.githubusercontent.com/opencloud-eu/awesome-apps/; manifest-src 'self'; media-src 'self'; object-src 'self' blob:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'
      │ Date: Mon, 03 Nov 2025 12:39:17 GMT
      │ Referrer-Policy: strict-origin-when-cross-origin
      │ Strict-Transport-Security: max-age=315360000; preload
      │ Vary: Origin
      │ X-Content-Type-Options: nosniff
      │ X-Frame-Options: SAMEORIGIN
      │ X-Permitted-Cross-Domain-Policies: none
      │ X-Request-Id: aaa0fe5e9d2a/MRPx71Ttfw-000046
      │ X-Robots-Tag: none
      │ X-Xss-Protection: 1; mode=block
      │ Body:
      │ string(0) ""
      │ 
      │ ### END RESPONSE
      │ 
      The content was expected to be 'contentnew', but actually is 'con'. HTTP status was 200
      Failed asserting that two strings are equal.
      --- Expected
      +++ Actual
      @@ @@
      -'contentnew'
      +'con'

rhafer avatar Nov 03 '25 12:11 rhafer

link to issue #1747

ScharfViktor avatar Nov 03 '25 12:11 ScharfViktor

To make things worse a similar metadata corruption can happen even with watchfs being disabled.

  1. User1 uploads a file file1.txt to a space
  2. User2 also uploads a file file1.txt to the same space
  3. the file uploaded by User1 gets into postprocessing first, but (for whatever reason) the postprocessing of the file uploaded by User2 finished before the postprocessing of User1's upload

Outcome:

  1. As soon as the postprocessing of the User2 upload finishes the TOO_EARLY status code vanishes from the propfind response (even though the postprocessing of the User1 upload is not finished)
  2. When postprocessing in finished for both files. The file ending up on the storage has the contents of the User1's upload, but the metadata attribute (blobid, size, ...) of the upload of User2. 💥
  3. A revision node exists with the metadata of User1's upload but without any content. The content uploaded by User2 is lost.

IMO the desired outcome should be:

  • There should be revision node created with the contents and metadata of the upload that finished postprocessing first.
  • The main filesystem node should have the content and metadata of the upload that finished postprocessing last.
  • The TOO_EARLY (425) response to only go away after the postprocessing finished for both uploads

rhafer avatar Nov 10 '25 17:11 rhafer