opencloud OpenCloud 3.7.0: Uploading a file and it got stuck with repeated logs

Describe the bug

When I upload a photo, it got stuck, never fully uploaded, and I can't use the file. There are repeated error log entries. It doesn't happen to every file I uploaded.

Steps to reproduce

Upload a photo

Expected behavior

The file is fully uploaded, and I can open or manage that file on OpenCloud webui

Actual behavior

The file was never fully uploaded. The file item is grey out. There are repeated error logs.

{"level":"warn","service":"thumbnails","method":"Thumbnails.GetThumbnail","duration":4.437716,"error":"{\"id\":\"eu.opencloud.api.thumbnails\",\"code\":425,\"detail\":\"File Processing\",\"status\":\"Too Early\"}","time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0/decorators/logging.go:46","message":"Failed to execute"}
{"level":"info","service":"proxy","proto":"HTTP/1.1","request-id":"bcb19a94-81eb-4eb4-836e-484447239fa9","traceid":"2d66caa222aefdce4901a6591e5cdecb","remote-addr":"171.97.34.29","method":"GET","status":425,"path":"/remote.php/dav/spaces/07230d49-6e02-4229-b4a5-2dacc7c40858$06953037-5f8b-404b-b0b1-f8d82fd39f16/.space/image.png","duration":13.914672,"bytes":170,"time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34","message":"access-log"}
{"level":"info","service":"auth-machine","host.name":"e7f828ff9836","pkg":"rgrpc","traceid":"0a2ce815120db54742ee8388d4c61353","time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/reva/[email protected]/internal/grpc/services/authprovider/authprovider.go:146","message":"user idp:\"https://opcloud.chotechai.com\" opaque_id:\"01c478ab-4494-4b57-9f18-f699ea872f43\" type:USER_TYPE_PRIMARY authenticated"}
{"level":"warn","service":"thumbnails","method":"Thumbnails.GetThumbnail","duration":3.230306,"error":"{\"id\":\"eu.opencloud.api.thumbnails\",\"code\":425,\"detail\":\"File Processing\",\"status\":\"Too Early\"}","time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0/decorators/logging.go:46","message":"Failed to execute"}
{"level":"info","service":"proxy","proto":"HTTP/1.1","request-id":"e6e432fe-344d-4fba-ac0a-3b15f0fc15e4","traceid":"0a2ce815120db54742ee8388d4c61353","remote-addr":"171.97.34.29","method":"GET","status":425,"path":"/remote.php/dav/spaces/07230d49-6e02-4229-b4a5-2dacc7c40858$9a0f8b6b-c836-4415-9372-8850e08ea288/wopi_diagram.png","duration":12.590062,"bytes":170,"time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34","message":"access-log"}
{"level":"info","service":"auth-machine","host.name":"e7f828ff9836","pkg":"rgrpc","traceid":"dca332b78751add35f05b2430f1ac0f9","time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/reva/[email protected]/internal/grpc/services/authprovider/authprovider.go:146","message":"user idp:\"https://opcloud.chotechai.com\" opaque_id:\"01c478ab-4494-4b57-9f18-f699ea872f43\" type:USER_TYPE_PRIMARY authenticated"}
{"level":"warn","service":"thumbnails","method":"Thumbnails.GetThumbnail","duration":3.552589,"error":"{\"id\":\"eu.opencloud.api.thumbnails\",\"code\":425,\"detail\":\"File Processing\",\"status\":\"Too Early\"}","time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/thumbnails/pkg/service/grpc/v0/decorators/logging.go:46","message":"Failed to execute"}
{"level":"info","service":"proxy","proto":"HTTP/1.1","request-id":"e6e432fe-344d-4fba-ac0a-3b15f0fc15e4","traceid":"dca332b78751add35f05b2430f1ac0f9","remote-addr":"171.97.34.29","method":"GET","status":425,"path":"/remote.php/dav/spaces/07230d49-6e02-4229-b4a5-2dacc7c40858$9a0f8b6b-c836-4415-9372-8850e08ea288/wopi_diagram.png","duration":12.38766,"bytes":170,"time":"2025-11-13T03:39:12Z","line":"github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34","message":"access-log"}

Setup

using stock docker-compose in opencloud-compose and modify stock .env for relevant server installation on Ubuntu.

## Basic Settings ##
# Define the docker compose log driver used.
# Defaults to local
LOG_DRIVER=
# If you're on an internet facing server, comment out following line.
# It skips certificate validation for various parts of OpenCloud and is
# needed when self signed certificates are used.
#INSECURE=true

## Features ##
# The following variable is a convenience variable to enable or disable features of this compose project.
# Example: if you want to use traefik and letsencrypt, you can set the variable to
#COMPOSE_FILE=docker-compose.yml:traefik/opencloud.yml
# This enables you to just run `docker compose up -d` and the compose files will be added to the stack.
# As alternative approach you can run `docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d`
# Default: OpenCloud and Collabora with traefik and letsencypt
# This needs DNS entries for the domain names used in the .env file.
#COMPOSE_FILE=docker-compose.yml:weboffice/collabora.yml:traefik/opencloud.yml:traefik/collabora.yml
# If you want to use the external proxy, you can use the following combination.
# DNS entries and certificates need to be managed by the external environment.
# The domain names need to be entered into the .env file.
##when using collabora
#COMPOSE_FILE=docker-compose.yml:search/tika.yml:radicale/radicale.yml:weboffice/collabora.yml:external-proxy/opencloud.yml:external-proxy/collabora.yml
##when using onlyoffice
COMPOSE_FILE=docker-compose.yml:search/tika.yml:radicale/radicale.yml:weboffice/onlyoffice.yml:external-proxy/opencloud.yml:external-proxy/onlyoffice.yml:antivirus/clamav.yml
# Keycloak Shared User Directory
#COMPOSE_FILE=docker-compose.yml:weboffice/collabora.yml:traefik/opencloud.yml:traefik/collabora.yml:idm/ldap-keycloak.yml:traefik/ldap-keycloak.yml

## Traefik Settings ##
# Note: Traefik is always enabled and can't be disabled.
# Serve Traefik dashboard.
# Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard.
# Defaults to "traefik.opencloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the traefik dashboard.
# Defaults to user "admin" and password "admin" (written as: "admin:$2y$05$KDHu3xq92SPaO3G8Ybkc7edd51pPLJcG1nWk3lmlrIdANQ/B6r5pq").
# To create user:password pair, it's possible to use this command:
# echo $(htpasswd -nB user) | sed -e s/\\$/\\$\\$/g
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates.
# Needs only be changed if this is a public facing server.
TRAEFIK_ACME_MAIL=
# Set to the following for testing to check the certificate process:
# "https://acme-staging-v02.api.letsencrypt.org/directory"
# With staging configured, there will be an SSL error in the browser.
# When certificates are displayed and are emitted by # "Fake LE Intermediate X1",
# the process went well and the envvar can be reset to empty to get valid certificates.
TRAEFIK_ACME_CASERVER=
# Enable the Traefik ACME (Automatic Certificate Management Environment) for automatic SSL certificate management.
TRAEFIK_SERVICES_TLS_CONFIG="tls.certresolver=letsencrypt"
# Enable Traefik to use local certificates.
#TRAEFIK_SERVICES_TLS_CONFIG="tls=true"
# You also need to provide a config file in ./config/traefik/dynamic/certs.yml
# Example:
# cat ./config/traefik/dynamic/certs.yml
# tls:
#   certificates:
#     - certFile: /certs/opencloud.test.crt
#       keyFile: /certs/opencloud.test.key
#       stores:
#         - default
#
# The certificates need to be copied into ./certs/, the absolute path inside the container is /certs/.
# You can also use TRAEFIK_CERTS_DIR=/path/on/host to set the path to the certificates directory.
# Enable the access log for Traefik by setting the following variable to true.
TRAEFIK_ACCESS_LOG=
# Configure the log level for Traefik.
# Possible values are "TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL" and "PANIC". Default is "ERROR".
TRAEFIK_LOG_LEVEL=


## OpenCloud Settings ##
# The opencloud container image.
# For production releases: "opencloudeu/opencloud"
# For rolling releases:    "opencloudeu/opencloud-rolling"
# Defaults to production if not set otherwise
OC_DOCKER_IMAGE=opencloudeu/opencloud-rolling
# The openCloud container version.
# Defaults to "latest" and points to the latest stable tag.
OC_DOCKER_TAG=
# Domain of openCloud, where you can find the frontend.
# Defaults to "cloud.opencloud.test"
OC_DOMAIN=opcloud.mydomain.com
# Demo users should not be created on a production instance,
# because their passwords are public. Defaults to "false".
# If demo users is set to "true", the following user accounts are created automatically:
# alan, mary, margaret, dennis and lynn - the password is 'demo' for all.
DEMO_USERS=
# Admin Password for the OpenCloud admin user.
# NOTE: This is only needed when using the built-in LDAP server (idm).
# If you are using an external LDAP server, the admin password is managed by the LDAP server.
# NOTE: This variable needs to be set before the first start of OpenCloud. Changes to this variable after the first start will be IGNORED.
# If not set, opencloud will not work properly. The container will be restarting.
# After the first initialization, the admin password can only be changed via the OpenCloud User Settings UI or by using the OpenCloud CLI.
# Documentation: https://docs.opencloud.eu/docs/admin/resources/common-issues#-change-admin-password-set-in-env
INITIAL_ADMIN_PASSWORD='ssd....8bKY'
# Define the openCloud loglevel used.
#
LOG_LEVEL=
# Define the kind of logging.
# The default log can be read by machines.
# Set this to true to make the log human readable.
# LOG_PRETTY=true
#
# Define the openCloud storage location. Set the paths for config and data to a local path.
# Ensure that the configuration and data directories are owned by the user and group with ID 1000:1000.
# This matches the default user inside the container and avoids permission issues when accessing files.
# Note that especially the data directory can grow big.
# Leaving it default stores data in docker internal volumes.
 OC_CONFIG_DIR=/home/ubuntu/opencloud/config
 OC_DATA_DIR=/home/ubuntu/opencloud/data
# OpenCloud Web can load extensions from a local directory.
# The default uses the bind mount to the config/opencloud/apps directory.
# Example: curl -L https://github.com/opencloud-eu/web-extensions/releases/download/unzip-v1.0.2/unzip-1.0.2.zip | tar -xz -C config/opencloud/apps
# NOTE: you need to restart the openCloud container to load the new extensions.
 OC_APPS_DIR=/home/ubuntu/opencloud/apps

# Define the ldap-server storage location. Set the paths for config and data to a local path.
# LDAP_CERTS_DIR=
# LDAP_DATA_DIR=

# S3 Storage configuration - optional
# OpenCloud supports S3 storage as primary storage.
# Per default, S3 storage is disabled and the decomposed storage driver is used.
# To enable S3 storage, add `storage/decomposeds3.yml` to the COMPOSE_FILE variable or to
# your startup command (`docker compose -f docker-compose.yml -f storage/decomposeds3.yml up`).
#
# Configure the S3 storage endpoint. Defaults to "http://minio:9000" for testing purposes.
DECOMPOSEDS3_ENDPOINT=
# S3 region. Defaults to "default".
DECOMPOSEDS3_REGION=
# S3 access key. Defaults to "opencloud"
DECOMPOSEDS3_ACCESS_KEY=
# S3 secret. Defaults to "opencloud-secret-key"
DECOMPOSEDS3_SECRET_KEY=
# S3 bucket. Defaults to "opencloud"
DECOMPOSEDS3_BUCKET=


# Define SMTP settings if you would like to send OpenCloud email notifications.
# To actually send notifications, you also need to enable the 'notifications' service
# by adding it to the START_ADDITIONAL_SERVICES variable below.
#
# NOTE: when configuring Inbucket, these settings have no effect, see inbucket.yml for details.
# SMTP host to connect to.
SMTP_HOST=mail.mydomain.com
# Port of the SMTP host to connect to.
SMTP_PORT=465
# An eMail address that is used for sending OpenCloud notification eMails
# like "opencloud notifications <[email protected]>".
SMTP_SENDER="opencloud notifications <[email protected]>"
# Username for the SMTP host to connect to.
SMTP_USERNAME="[email protected]"
# Password for the SMTP host to connect to.
SMTP_PASSWORD="&u....jQm"
# Authentication method for the SMTP communication.
SMTP_AUTHENTICATION=auto
# Encryption method for the SMTP communication. Possible values are 'starttls', 'ssltls' and 'none'
SMTP_TRANSPORT_ENCRYPTION=ssltls
# Allow insecure connections to the SMTP server. Defaults to false.
SMTP_INSECURE=

# Additional services to be started on opencloud startup
# The following list of services is not started automatically and must be
# manually defined for startup:
# IMPORTANT: Add any services to the startup list comma separated like "notifications,antivirus" etc.
START_ADDITIONAL_SERVICES="notifications"


## Default Enabled Services ##

### Apache Tika Content Analysis Toolkit ###
# Tika (search) is disabled by default due to performance reasons.
# Tika is used to extract metadata and text from various file formats.
# Enable it by adding the following to the COMPOSE_FILE variable:
# search/tika.yml or by using the following command:
# docker compose -f docker-compose.yml -f search/tika.yml up -d
# Set the desired docker image tag or digest.
# Defaults to "apache/tika:latest-full"
TIKA_IMAGE=

### IMPORTANT Note for Online Office Apps ###
# To avoid app interlocking issues, you should select only one app to be active/configured.
# This is due the fact that there is currently no app interlocking for the same file and one
# has to wait for a lock release to open the file with another app.

### Collabora Settings ###
# Domain of Collabora, where you can find the frontend.
# Defaults to "collabora.opencloud.test"
#COLLABORA_DOMAIN="opcollabora.mydomain.com"
COLLABORA_DOMAIN="onlyoffice.mydomain.com"
# Domain of the wopiserver which handles Collabora.
# Defaults to "wopiserver.opencloud.test"
WOPISERVER_DOMAIN="opwopi.mydomain.com"
# Admin user for Collabora.
# Defaults to "admin".
# Collabora Admin Panel URL:
# https://{COLLABORA_DOMAIN}/browser/dist/admin/admin.html
COLLABORA_ADMIN_USER="admin"
# Admin password for Collabora.
# Defaults to "admin".
COLLABORA_ADMIN_PASSWORD="DE...td8"
# Set to true to enable SSL handling in Collabora Online, this is only required if you are not using a reverse proxy.
# Default is true if not specified.
COLLABORA_SSL_ENABLE=false
# If you're on an internet-facing server, enable SSL verification for Collabora Online.
# Please comment out the following line:
#COLLABORA_SSL_VERIFICATION=false


### Virusscanner Settings ###
# IMPORTANT: If you enable antivirus, you also MUST configure the START_ADDITIONAL_SERVICES
# envvar in the OpenCloud Settings above by adding 'antivirus' to the list.
# The maximum scan size the virus scanner can handle, needs adjustment in the scanner config as well.
# Usable common abbreviations: [KB, KiB, MB, MiB, GB, GiB, TB, TiB, PB, PiB, EB, EiB], example: 2GB.
# Defaults to "100MB"
#ANTIVIRUS_MAX_SCAN_SIZE=
# Usable modes: partial, skip.
# Defaults to "partial"
#ANTIVIRUS_MAX_SCAN_SIZE_MODE=
# Image version of the ClamAV container.
# Defaults to "latest"
CLAMAV_DOCKER_TAG=1.4 #arm64 latest is 1.4 and no latest tag


### Inbucket Settings ###
# Inbucket is a mail catcher tool for testing purposes.
# DO NOT use in Production.
# email server (in this case inbucket acts as mail catcher).
# Domain for Inbucket. Defaults to "mail.opencloud.test".
INBUCKET_DOMAIN=


### Compose Configuration ###
# Path separator for supplemental compose files specified in COMPOSE_FILE.
COMPOSE_PATH_SEPARATOR=:

### Ldap Settings ###
# LDAP is always needed for OpenCloud to store user data as there is no relational database.
# The built-in LDAP server should used for testing purposes or small installations only.
# For production installations, it is recommended to use an external LDAP server.
# We are using OpenLDAP as the default LDAP server because it is proven to be stable and reliable.
# This LDAP configuration is known to work with OpenCloud and provides a blueprint for
# configuring an external LDAP server based on other products like Microsoft Active Directory or other LDAP servers.
#
# Password of LDAP bind user "cn=admin,dc=opencloud,dc=eu". Defaults to "admin"
LDAP_BIND_PASSWORD=
# The LDAP server also creates an openCloud admin user dn: uid=admin,ou=users,dc=opencloud,dc=eu
# The initial password for this user is "admin"
# NOTE: This password can only be set once, if you want to change it later, you have to use the OpenCloud User Settings UI.
# If you changed the password and lost it, you need to execute the following LDAP query to reset it:
# enter the ldap-server container with `docker compose exec ldap-server sh`
# and run the following command to change the password:
# ldappasswd -H ldap://127.0.0.1:1389 -D "cn=admin,dc=opencloud,dc=eu" -W "uid=admin,ou=users,dc=opencloud,dc=eu"
# You will be prompted for the LDAP bind password.
# The output should provide you a new password for the admin user.


### Keycloak Settings ###
# Keycloak is an open-source identity and access management solution.
# We are using Keycloak as the default identity provider on production installations.
# It can be used to federate authentication with other identity providers like
# Microsoft Entra ID, ADFS or other SAML/OIDC providers.
# The use of Keycloak as bridge between OpenCloud and other identity providers creates more control over the
# authentication process, the allowed clients and the session management.
# Keycloak also manages the Role Based Access Control (RBAC) for OpenCloud.
# Keycloak can be used in two different modes:
# 1. Autoprovisioning: New users are automatically created in openCloud when they log in for the first time.
# 2. Shared User Directory: Users are created in Keycloak and can be used in OpenCloud immediately
# because the LDAP server is connected to both Keycloak and OpenCloud.
# Only use one of the two modes at a time.

## Autoprovisioning Mode ##
# Use together with idm/external-idp.yml
# If you want to use a keycloak for local testing, you can use testing/external-keycloak.yml and testing/ldap-manager.yml
# Domain of your Identity Provider.
IDP_DOMAIN=
# IdP Issuer URL, which is used to identify the Identity Provider.
# We need the complete URL, including the protocol (http or https) and the realm.
# Example: "https://keycloak.opencloud.test/realms/openCloud"
IDP_ISSUER_URL=
# Url of the account edit page from your Identity Provider.
IDP_ACCOUNT_URL=

## Shared User Directory Mode ##
# Use together with idm/ldap-keycloak.yml and traefik/ldap-keycloak.yml
# Domain for Keycloak. Defaults to "keycloak.opencloud.test".
KEYCLOAK_DOMAIN=
# Admin user login name. Defaults to "kcadmin".
KEYCLOAK_ADMIN=
# Admin user login password. Defaults to "admin".
KEYCLOAK_ADMIN_PASSWORD=
# Keycloak Database username. Defaults to "keycloak".
KC_DB_USERNAME=
# Keycloak Database password. Defaults to "keycloak".
KC_DB_PASSWORD=

### Radicale Setting ###
# Radicale is a small open-source CalDAV (calendars, to-do lists) and CardDAV (contacts) server.
# When enabled OpenCloud is configured as a reverse proxy for Radicale, providing all authenticated
# OpenCloud users access to a Personal Calendar and Addressbook
# Docker image to use for the Radicale Container
RADICALE_DOCKER_IMAGE=opencloudeu/radicale
# Docker tag to pull for the Radicale Container
RADICALE_DOCKER_TAG=latest
# Define the storage location for the Radicale data. Set the path to a local path.
# Ensure that the configuration and data directories are owned by the user and group with ID 1000:1000.
# This matches the default user inside the container and avoids permission issues when accessing files.
# Leaving it default stores data in docker internal volumes.
RADICALE_DATA_DIR=/home/ubuntu/radicale/data

Additional context

The issue doesn't happen to every file I uploaded, but it happened quite often. Sometimes I have to reinstall OpenCloud from scratch to get rid of many dangled files like this.

Nov 13 '25 04:11 europacafe

Sometimes I have to reinstall OpenCloud from scratch to get rid of many dangled files like this.

Sounds scary😱. Better fix the issue. There are maintenance commands to restart postprocessing.

The logs that you provide do not point to the error. The error with the thumbnails is caused by a file where the upload postprocessing did not finish. We need the logs during the upload.

Multiple reasons could be the cause:

Reverse proxy (which one are you using?)
Storage and permissions
virus scanning

Nov 13 '25 06:11 micbar

Thanks.

As discussed in length in the discussion board https://github.com/orgs/opencloud-eu/discussions/1852 , here is the conclusion:

I found the culprit is the clamav. Once I removed clamav and ran restart again, all the stuck files came through. The error log, as posted above, "425" "Too Early" seems to indicate the uploaded files were stuck after being sent to claimav for scan.

As I run OpenCloud on Oracle Cloud "arch64" ubuntu, I have to use clamav arm64 v1.4 (its latest). I'm not sure it was due to the arm version.

Nov 14 '25 01:11 europacafe

After reading this I rolled back to 3.5.0 and after that I have no issues uploading files. Mine (file uploads) were stuck in 'postprocessing' state

I'm using the most basic docker run setup without compose, just build the container and run it with ocker run --name opencloud --rm -d -p 9200:9200 -v $HOME/opencloud/opencloud-config:/etc/opencloud -v $HOME/opencloud/opencloud-data:/var/lib/opencloud -e OC_INSECURE=true -e PROXY_HTTP_ADDR=0.0.0.0:9200 -e OC_URL=https://192.168.1.3:9200 opencloudeu/opencloud:opencloud_release_crypt_350

@europacafe if you order your files by date it will likely show these uploaded before 3.7.0 as it should and after - as 'postprocessing' I can't tell which commit from 3.7.0 is causing this, but the logs were showing these:


{"level":"error","service":"postprocessing","uploadID":"","error":"Failed to get object from bucket: nats: invalid key","time":"2025-11-14T15:49:55Z","message":"cannot get upload"}
{"level":"error","service":"postprocessing","error":"event error: cannot get upload","time":"2025-11-14T15:49:55Z","message":"continuing"}

and more

 {"level":"error","service":"postprocessing","uploadID":"","error":"Failed to get object from bucket: nats: invalid key","time":"2025-11-13T16:59:10Z","message":"cannot get upload"}
{"level":"error","service":"postprocessing","error":"event error: cannot get upload","time":"2025-11-13T16:59:10Z","message":"continuing"}
{"level":"error","service":"thumbnails","time":"2025-11-13T17:00:09Z","message":"resource info is missing checksum"}
{"level":"error","spaceid":"ed350ee9-4403-4ff4-b958-5be63d7f0c48","nodeid":"8dc4adc3-e717-4e26-8052-a5c41723d0a5","error":"failed to upload file to blobstore: failed to set xattr 'security.SMACK64' on temp file '/var/lib/opencloud/storage/users/users/2ad53bfe-6565-4487-b929-2184b0083610/.oc-tmp/20765ea2-0f48-4e4a-bbbf-36b4ddf884c2' - xattr.Set /var/lib/opencloud/storage/users/users/2ad53bfe-6565-4487-b929-2184b0083610/.oc-tmp/20765ea2-0f48-4e4a-bbbf-36b4ddf884c2 security.SMACK64: operation not permitted","time":"2025-11-13T17:00:17Z","message":"could not finalize upload"}
{"level":"error","service":"proxy","error":"could not get user by claim username with value admin: rpc error: code = Canceled desc = context canceled","time":"2025-11-13T17:11:54Z","message":"Could not get user by claim"}
{"level":"error","service":"proxy","error":"{\"id\":\"go.micro.client\",\"code\":408,\"detail\":\"context canceled\",\"status\":\"Request Timeout\"}","time":"2025-11-13T17:11:54Z","message":"Could not load roles"}
{"level":"error","service":"proxy","error":"{\"id\":\"go.micro.client\",\"code\":408,\"detail\":\"context canceled\",\"status\":\"Request Timeout\"}","time":"2025-11-13T17:11:54Z","message":"Could not get user roles"}
{"level":"error","service":"gateway","host.name":"9ffd89adc416","pkg":"rgrpc","traceid":"fa422fdef09c83400f5a1b48c6150407","user-agent":"grpc-go/1.76.0","from":"tcp://127.0.0.1:35128","uri":"/cs3.gateway.v1beta1.GatewayAPI/GetUserByClaim","start":"13/Nov/2025:17:11:54 +0000","end":"13/Nov/2025:17:11:54 +0000","time_ns":8688734,"code":"Canceled","time":"2025-11-13T17:11:54Z","message":"gateway: error calling GetUserByClaim: rpc error: code = Canceled desc = context canceled"}
2025/11/13 18:04:10 http: TLS handshake error from 192.168.1.2:52202: EOF
2025/11/13 18:04:10 http: TLS handshake error from 192.168.1.2:52203: EOF
2025/11/13 18:04:40 http: TLS handshake error from 192.168.1.2:52210: remote error: tls: unknown certificate
2025/11/13 18:04:40 http: TLS handshake error from 192.168.1.2:52212: remote error: tls: unknown certificate

Nov 14 '25 16:11 mrodozov

According to @europacafe it has nothing to do with the openCloud version.

His clamav was disfunctional.

Nov 14 '25 16:11 micbar

@rhafer @butonic That xattr.Set looks weird. Do we set xattrs in the security namespace?

Nov 14 '25 16:11 micbar

invalid

I'm facing the same issue with exact same logs:

opencloud       | {"level":"error","spaceid":"f4fe262b-755d-488c-a26f-5807e8f18f86","nodeid":"bc8303f8-5832-42c5-b186-bc9c0460c75f","error":"failed to upload file to blobstore: failed to move temp file '/var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d' to node '/var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg' - rename /var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d /var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg: invalid cross-device link","time":"2025-11-17T22:46:29Z","message":"could not finalize upload"}

and a neverending following logs

opencloud       | 2025-11-17T22:46:30Z ERR cannot get upload error="Failed to get object from bucket: nats: invalid key" line=github.com/opencloud-eu/opencloud/services/postprocessing/pkg/service/service.go:261 service=postprocessing uploadID=
opencloud       | 2025-11-17T22:46:30Z ERR continuing error="event error: cannot get upload" line=github.com/opencloud-eu/opencloud/services/postprocessing/pkg/service/service.go:140 service=postprocessing

Nov 17 '25 22:11 carlosjfcasero

@rhafer @butonic That xattr.Set looks weird. Do we set xattrs in the security namespace?

No. But in https://github.com/opencloud-eu/reva/blob/main/pkg/storage/fs/posix/blobstore/blobstore.go#L93 we try to copy all extended attributes between nodes. The security.SMACK attribute, and other,s is something maintained by smack ("Simplified Mandatory Access Control Kernel', some "mandatory access control" framwork for linux). This is likely some fallout of https://github.com/opencloud-eu/reva/commit/e65b97c25d9a0fc94320134bd0085e1575dcbeed.

Nov 18 '25 13:11 rhafer

I'm facing the same issue with exact same logs:

opencloud       | {"level":"error","spaceid":"f4fe262b-755d-488c-a26f-5807e8f18f86","nodeid":"bc8303f8-5832-42c5-b186-bc9c0460c75f","error":"failed to upload file to blobstore: failed to move temp file '/var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d' to node '/var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg' - rename /var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d /var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg: invalid cross-device link","time":"2025-11-17T22:46:29Z","message":"could not finalize upload"}

While the error messsage is similar. This problem itself is differernt. @carlosjfcasero could you please open a separate bug report for that? While at that, please provide info (in the new issue) about your filesystem layout e.g. did you mount var/lib/opencloud/storage/users/projects/F3/.oc-tmp to a separate volume?

Nov 18 '25 13:11 rhafer

I'm facing the same issue with exact same logs:
opencloud       | {"level":"error","spaceid":"f4fe262b-755d-488c-a26f-5807e8f18f86","nodeid":"bc8303f8-5832-42c5-b186-bc9c0460c75f","error":"failed to upload file to blobstore: failed to move temp file '/var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d' to node '/var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg' - rename /var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d /var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg: invalid cross-device link","time":"2025-11-17T22:46:29Z","message":"could not finalize upload"}
While the error messsage is similar. This problem itself is differernt. @carlosjfcasero could you please open a separate bug report for that? While at that, please provide info (in the new issue) about your filesystem layout e.g. did you mount var/lib/opencloud/storage/users/projects/F3/.oc-tmp to a separate volume?

As far as I can see, my logs are exactly the same as @mrodozov 's. If you want, for sure I can open a different ticket @rhafer

Nov 18 '25 21:11 carlosjfcasero

As far as I can see, my logs are exactly the same as @mrodozov 's

It's really different:

{".... failed to move temp file '/var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d' to node '/var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg' - rename /var/lib/opencloud/storage/users/projects/F3/.oc-tmp/cb3ec316-c836-4799-bc81-5359e8cc343d /var/lib/opencloud/storage/users/projects/F3/Fotos y Vídeos/2025/navidad 2025.jpg: invalid cross-device link"

vs

{"failed to set xattr 'security.SMACK64' on temp file '/var/lib/opencloud/storage/users/users/2ad53bfe-6565-4487-b929-2184b0083610/.oc-tmp/20765ea2-0f48-4e4a-bbbf-36b4ddf884c2' - xattr.Set /var/lib/opencloud/storage/users/users/2ad53bfe-6565-4487-b929-2184b0083610/.oc-tmp/20765ea2-0f48-4e4a-bbbf-36b4ddf884c2 security.SMACK64: operation not permitted"

So yes, please open a separate issue.

Nov 20 '25 08:11 rhafer