server [Bug]: Drag and Drop folder failed in a load-balanced System

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

We have noticed that in a load-balanced environment without Sticky Sessions, dragging and dropping a folder containing multiple files no longer works. In the case of an environment with 2 application servers, only half of the files are uploaded correctly. The web interface displays error messages saying "Operation blocked by access control."

We can see that one of the nodes is returning 403 errors on file uploads while the second one handles them correctly. Based on repeated tests, it's clear that this issue is not specific to any particular node. When we isolate either node 1 or node 2, drag and drop functionality works.

For your information, this works perfectly in 27.1.4 under the same conditions.

Steps to reproduce

load-balanced environment: 1 reverse proxy, 2 application servers, 1 MariaDB server + Redis
configure the reverse proxy to load-balance requests in a round-robin fashion across the 2 application servers without sticky sessions
configure PHP to store sessions in Redis
configure Nextcloud with local memcache on APCu and distributed memcache on Redis
Drag and Drop a folder (testDrag) which contain 10 files (1 to 10)

Expected behavior

The folder and all files should be uploaded without any error messages.

Installation method

Community Manual installation with Archive

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[ ] Default user-backend (database)
[X] LDAP/ Active Directory
[X] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "dbtype": "mysql",
        "version": "29.0.3.4",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "ldapUserCleanupInterval": 0,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "dbindex": 1
        },
        "filesystem_check_changes": 0,
        "enable_avatars": false,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "share_folder": "\/Shared",
        "cron_log": true,
        "loglevel": "2",
        "maintenance": false,
        "trashbin_retention_obligation": 30,
        "trashbin_auto_expire": true,
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "knowledgebaseenabled": false,
        "sharing.maxAutocompleteResults": 20,
        "sharing.minSearchStringLength": 3,
        "default_phone_region": "FR",
        "maintenance_window_start": 3,
        "activity_expire_days": "180",
        "simpleSignUpLink.shown": false,
        "default_timezone": "Europe\/Paris",
        "logtimezone": "Europe\/Paris",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\MP3",
            "OC\\Preview\\PNG",
            "OC\\Preview\\XBitmap"
        ]
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contactsinteraction: 1.10.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - lookup_server_connector: 1.17.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - photos: 2.5.0
  - provisioning_api: 1.19.0
  - richdocuments: 8.4.3
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - user_ldap: 1.20.0
  - user_saml: 6.1.3
  - viewer: 2.3.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - circles: 29.0.0-dev (installed 25.0.0)
  - dashboard: 7.9.0 (installed 7.0.0)
  - encryption: 2.17.0
  - federation: 1.19.0 (installed 1.3.0)
  - files_external: 1.21.0
  - files_reminders: 1.2.0 (installed 1.0.0)
  - firstrunwizard: 2.18.0 (installed 2.14.0)
  - logreader: 2.14.0 (installed 2.10.0)
  - nextcloud_announcements: 1.18.0 (installed 1.14.0)
  - password_policy: 1.19.0 (installed 1.15.0)
  - privacy: 1.13.0 (installed 1.9.0)
  - recommendations: 2.1.0 (installed 1.4.0)
  - related_resources: 1.4.0 (installed 1.0.4)
  - support: 1.12.0 (installed 1.8.0)
  - survey_client: 1.17.0 (installed 1.13.0)
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0 (installed 1.3.0)
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1 (installed 1.17.0)
  - user_status: 1.9.0 (installed 1.0.1)
  - weather_status: 1.9.0 (installed 1.7.0)

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

Browser log (Firefox 127.0):

[DEBUG] files: Initial templates folder 
Object { app: "files", uid: "***LOGIN***", level: "2", templatesPath: false }
ConsoleLogger.js:65:18
[DEBUG] files: FileActions initialized 
Object { app: "files", uid: "***LOGIN***", level: "2" }
index.mjs:45:16
[DEBUG] files: NewFileMenu initialized 
Object { app: "files", uid: "***LOGIN***", level: "2" }
index.mjs:45:16
[DEBUG] files: Generating favorites view 
Object { app: "files", uid: "***LOGIN***", level: "2", favoriteFolders: [] }
ConsoleLogger.js:65:18
[DEBUG] files: Navigation service initialized 
Object { app: "files", uid: "***LOGIN***", level: "2" }
index.mjs:45:16
[INFO] activity: Activity API registered 
Object { app: "activity", uid: "***LOGIN***", level: "2" }
logger-BVTvLOjn.mjs:2:30195
Unified search initialized! UnifiedSearch.vue:54
OCA.Files.Settings initialized Settings.js:29:10
[DEBUG] files: Navigation mounted. Showing requested view 
Object { app: "files", uid: "***LOGIN***", level: "2", view: {…} }
ConsoleLogger.js:65:18
[DEBUG] uploader: Destination set 
Object { app: "uploader", uid: "***LOGIN***", level: "2", folder: {…} }
ConsoleLogger.js:65:18
[DEBUG] uploader: Upload workspace initialized 
Object { app: "uploader", uid: "***LOGIN***", level: "2", destination: {…}, root: "https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***", isPublic: false, maxChunksSize: 524288000 }
ConsoleLogger.js:65:18
OCA.Files.Sidebar initialized Sidebar.js:36:10
OCA.Viewer initialized viewer-main.mjs:2187:581
OCA.Sharing.ShareSearch initialized ShareSearch.js:33:10
OCA.Sharing.ExternalLinkActions initialized ExternalLinkActions.js:33:10
OCA.Sharing.ExternalShareActions initialized ExternalShareActions.js:33:10
OCA.Comments.View initialized comments-app.js:32:8
[INFO] files: Initializing unified search plugin: folder search from files app 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:68:18
[INFO] viewer: 6 viewer handlers registered 
Object { app: "viewer", uid: "***LOGIN***", level: "2", handlers: (6) […] }
viewer-main.mjs:44:184629
session heartbeat polling started session-heartbeat.js:103:9
[DEBUG] activity: Registered new sidebar action 
Object { app: "activity", uid: "***LOGIN***", level: "2" }
logger-BVTvLOjn.mjs:2:30110
[DEBUG] activity: Registered new sidebar actions factory 
Object { app: "activity", uid: "***LOGIN***", level: "2" }
logger-BVTvLOjn.mjs:2:30110
[INFO] comments: Comments plugin registered for Activity sidebar action 
Object { app: "comments", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:68:18
Registering notifications container as a menu NotificationsApp.vue:237
Notifications permissions denied NotificationsApp.vue:486
Polling interval updated to 30000 NotificationsApp.vue:443
Started background fetcher as session_keepalive is enabled NotificationsApp.vue:293
[DEBUG] core: this browser is officially supported ! 🚀 
Object { app: "core", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
Search providers 
Array(5) [ {…}, {…}, {…}, {…}, {…} ]
UnifiedSearchModal.vue:275
Contacts 
Array [ {…} ]
UnifiedSearchModal.vue:279
Got notification data, restoring default polling interval. NotificationsApp.vue:399
[DEBUG] files: SW registered:  
Object { app: "files", uid: "***LOGIN***", level: "2", registration: ServiceWorkerRegistration }
ConsoleLogger.js:65:18
[DEBUG] files: Fetched contents 
Object { app: "files", uid: "***LOGIN***", level: "2", dir: "/", folder: {…}, contents: (8) […] }
ConsoleLogger.js:65:18
[DEBUG] files: Directory contents changed 
Object { app: "files", uid: "***LOGIN***", level: "2", view: {…}, folder: {…}, contents: (8) […] }
ConsoleLogger.js:65:18
[DEBUG] files: FileListHeaders initialized 
Object { app: "files", uid: "***LOGIN***", level: "2" }
index.mjs:45:16
[DEBUG] uploader: Destination set 
Object { app: "uploader", uid: "***LOGIN***", level: "2", folder: {…} }
ConsoleLogger.js:65:18
[DEBUG] uploader: UploadPicker initialised 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: Cancelled previous ongoing fetch 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: Fetched contents 
Object { app: "files", uid: "***LOGIN***", level: "2", dir: "/", folder: {…}, contents: (8) […] }
ConsoleLogger.js:65:18
[DEBUG] files: Directory contents changed 
Object { app: "files", uid: "***LOGIN***", level: "2", view: {…}, folder: {…}, contents: (8) […] }
ConsoleLogger.js:65:18
[DEBUG] uploader: Destination set 
Object { app: "uploader", uid: "***LOGIN***", level: "2", folder: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Updating selection 
Object { app: "files", uid: "***LOGIN***", level: "2", selection: (1) […] }
ConsoleLogger.js:65:18
[DEBUG] files: Directory contents changed 
Object { app: "files", uid: "***LOGIN***", level: "2", view: {…}, folder: {…}, contents: (7) […] }
ConsoleLogger.js:65:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: Handling recursive file tree 
Object { app: "files", uid: "***LOGIN***", level: "2", entry: "testDrag" }
ConsoleLogger.js:65:18
[DEBUG] files: Dropped 
Object { app: "files", uid: "***LOGIN***", level: "2", event: drop, folder: {…}, fileTree: File }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading files to / 
Object { app: "files", uid: "***LOGIN***", level: "2", root: File, contents: (1) […] }
ConsoleLogger.js:65:18
Processing directory 
Object { relativePath: "/testDrag" }
DropService.ts:121:28
XHRPROPFIND
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag
[HTTP/1.1 404 Not Found 30ms]

[DEBUG] files: Directory does not exist, creating it 
Object { app: "files", uid: "***LOGIN***", level: "2", absolutePath: "/files/***LOGIN***/testDrag" }
ConsoleLogger.js:65:18
XHRPROPFIND
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag
[HTTP/1.1 404 Not Found 28ms]

[DEBUG] files: Directory contents changed 
Object { app: "files", uid: "***LOGIN***", level: "2", view: {…}, folder: {…}, contents: (8) […] }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/4 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 4 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/4 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/10 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 10 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/10 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/2 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 2 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/2 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/3 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 3 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/3 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/1 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 1 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/1 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/9 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 9 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/9 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/6 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 6 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/6 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/7 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 7 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/7 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/5 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 5 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/5 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[DEBUG] files: Uploading file to /testDrag/8 
Object { app: "files", uid: "***LOGIN***", level: "2", file: File }
ConsoleLogger.js:65:18
[DEBUG] uploader: Uploading 8 to https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/8 
Object { app: "uploader", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] uploader: Initializing regular upload 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/4
[HTTP/1.1 403 Forbidden 28ms]

XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/2
[HTTP/1.1 403 Forbidden 44ms]

[ERROR] uploader: Failed uploading 4 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/3
[HTTP/1.1 403 Forbidden 26ms]

[ERROR] uploader: Failed uploading 2 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
    value ConsoleLogger.js:74
    value ConsoleLogger.js:100
    i index-DM2X1kc6.mjs:275
    add index.js:180
    #x index.js:118
    #L index.js:149
    start index.js:218
    start index-DM2X1kc6.mjs:223
    Us DropService.ts:143
    onDrop DragAndDropNotice.vue:120
    VueJS 27
[ERROR] uploader: Failed uploading 3 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
[DEBUG] uploader: Successfully uploaded 10 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/6
[HTTP/1.1 403 Forbidden 28ms]

[ERROR] uploader: Failed uploading 6 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
[DEBUG] uploader: Successfully uploaded 1 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/5
[HTTP/1.1 403 Forbidden 32ms]

[DEBUG] uploader: Successfully uploaded 9 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
XHRPUT
https://***URL-PLATEFORME***/remote.php/dav/files/***LOGIN***/testDrag/8
[HTTP/1.1 403 Forbidden 25ms]

[ERROR] uploader: Failed uploading 5 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
[ERROR] uploader: Failed uploading 8 
Object { app: "uploader", uid: "***LOGIN***", level: "2", error: {…}, file: File, upload: {…} }
ConsoleLogger.js:74:18
[DEBUG] uploader: Successfully uploaded 7 
Object { app: "uploader", uid: "***LOGIN***", level: "2", file: File, upload: {…} }
ConsoleLogger.js:65:18
[ERROR] files: Error while uploading files 
Object { app: "files", uid: "***LOGIN***", level: "2", errors: (6) […] }
ConsoleLogger.js:74:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18
[DEBUG] files: VirtualList: resizeObserver updated 
Object { app: "files", uid: "***LOGIN***", level: "2" }
ConsoleLogger.js:65:18

Apache2 first node:

192.168.1.1 - - [27/Jun/2024:12:22:51 +0200] "PROPFIND /remote.php/dav/files/***LOGIN***/testDrag HTTP/1.1" 404 973 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:51 +0200] "PROPFIND /remote.php/dav/files/***LOGIN*** HTTP/1.1" 207 1178 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PROPFIND /remote.php/dav/files/***LOGIN***/testDrag HTTP/1.1" 207 1498 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/4 HTTP/1.1" 403 736 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/3 HTTP/1.1" 403 736 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/9 HTTP/1.1" 403 736 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/6 HTTP/1.1" 403 736 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/8 HTTP/1.1" 403 736 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

Apache2 second node:

192.168.1.1 - - [27/Jun/2024:12:22:51 +0200] "PROPFIND /remote.php/dav/files/***LOGIN***/ HTTP/1.1" 207 2249 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:51 +0200] "PROPFIND /remote.php/dav/files HTTP/1.1" 207 1056 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PROPFIND /remote.php/dav/files/***LOGIN***/testDrag HTTP/1.1" 207 1180 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "GET /index.php/apps/files/api/v1/stats HTTP/1.1" 200 973 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/10 HTTP/1.1" 204 686 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/2 HTTP/1.1" 201 742 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/1 HTTP/1.1" 204 686 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/7 HTTP/1.1" 204 686 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:52 +0200] "PUT /remote.php/dav/files/***LOGIN***/testDrag/5 HTTP/1.1" 201 742 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
192.168.1.1 - - [27/Jun/2024:12:22:57 +0200] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 758 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

Jun 27 '24 10:06 dsisysteme

The issue is still present in 29.0.4

Jul 24 '24 12:07 dsisysteme

Hi,

Can you share the exact 403 error from the logs please? I'm guessing your two instances share the same DB and data folder?

Jul 26 '24 14:07 skjnldsv

Hi,

Could you specify the information you are missing regarding the apache2 logs we included in the issue description?

Regarding the second question, indeed, the two instances share the same database and the same data folder. We also have a Redis instance that stores PHP sessions and distributed cache.

Jul 26 '24 14:07 dsisysteme

I meant the content of the 403 response. Either the json entry in nextcloud.log if any, or the browser network request response.

Jul 26 '24 16:07 skjnldsv

On a slight different note, can you share how you configured/enabled your shared session ?

Jul 26 '24 16:07 skjnldsv

Unfortunately, there is no log in the nextcloud.log. Here is an image of the sequence of requests:

All PUT requests with a 201 status are handled by node 2, and the 403 errors are on node 1. It seems that the node which handled the MKCOL request is the one where the PUT requests have a 201 status.

here is the extract of the responses from the requests at the browser level: PUT 403:

HTTP/1.1 403 Forbidden
Date: Mon, 29 Jul 2024 13:37:03 GMT
Server: Apache
Strict-Transport-Security: max-age=15768000
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: noindex, nofollow
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'none';
Content-Type: application/xml; charset=utf-8
Keep-Alive: timeout=5, max=92
Connection: Keep-Alive
Transfer-Encoding: chunked

PUT 201:

HTTP/1.1 201 Created
Date: Mon, 29 Jul 2024 13:37:03 GMT
Server: Apache
Strict-Transport-Security: max-age=15768000
Referrer-Policy: no-referrer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Robots-Tag: noindex, nofollow
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'none';
X-OC-MTime: accepted
OC-FileId: 250391672oc7elpk9mous
X-Request-Id: rz2VVlqVxSm4UqXra3xI
OC-ETag: "c1677a26a6ae01fc281bb0a4905400a3"
X-Debug-Token: rz2VVlqVxSm4UqXra3xI
ETag: "c1677a26a6ae01fc281bb0a4905400a3"
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

For PHP sessions, we use the following configuration at the fpm pool level on the 2 nodes:

php_admin_value[session.save_path] = tcp://<redis-server>:6379
php_admin_value[session.save_handler] = redis

For nextcloud, we force the distributed cache to index 1 to avoid conflicts. "dbindex": 1

Jul 29 '24 14:07 dsisysteme

here is the extract of the responses from the requests at the browser level: PUT 403:

Please also give us the xml body response, not only the headers

Jul 30 '24 06:07 skjnldsv

Sorry for the misunderstanding.

Here is the XML body of the 403 response:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\Forbidden</s:exception>
  <s:message/>
</d:error>

Jul 30 '24 07:07 dsisysteme

The issue is still present in 29.0.5

Aug 29 '24 13:08 dsisysteme

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

Oct 01 '24 00:10 nextcloud-command

This problem is still present in version 30.0.0. In a load-balanced environment without Sticky Sessions, dragging and dropping a folder containing multiple files no longer works, as well as the upload folder button.

Oct 01 '24 05:10 dsisysteme

In a load-balanced environment without Sticky Sessions, dragging and dropping a folder containing multiple files no longer works, as well as the upload folder button.

Works for me with multiple instances in a cluster and sticky sessions. What reverse proxy are you using? Can you maybe add a custom header for each apache node to make sure the sticky sessions are really working?

Oct 11 '24 14:10 skjnldsv

The problem occurs when the reverse proxy is configured without sticky sessions. We described our architecture (reverse proxy, Redis, session) in the issue description. Indeed, it works with sticky sessions, but this is not the most optimized mode for load balancing. As mentioned in the issue description, it worked perfectly in version 27.1.4.

Oct 11 '24 14:10 dsisysteme

For PHP sessions, we use the following configuration at the fpm pool level on the 2 nodes:

What about your Redis session locking configuration?

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/caching_configuration.html#using-the-redis-session-handler

Oct 29 '24 17:10 joshtrichards

We do have these configurations enabled.

We tried without these settings and have the same behavior in both cases.

Oct 31 '24 09:10 dsisysteme

As mentioned in the issue description, it worked perfectly in version 27.1.4.

Weird, I was never able to make it work on any previous versions of Nextcloud. And we haven't changed any backend implementation of chunk upload on 28 and above 🤔

Nov 08 '24 07:11 skjnldsv

Fixed!

For anyone finding this, and if you also experience one node failing but not the other, this is most likely a data issue. You need to make sure your webserver cluster shares the same data directory and that your storage layer doesn't have any caching. Here, the NFS storage had attributes caching enabled, you need the noac option to make it work

The "noac" option is a client side option available that disables attribute caching.

We'll try to improve and see if/how we can catch that from nextcloud system checks

Jan 30 '25 14:01 skjnldsv

After some testing, it seems more interesting to set the option lookupcache=none on the NFS client.

The noac option significantly impacts performance.

Jan 30 '25 14:01 dsisysteme