opencloud-eu
Some files corrupted on upload, others stuck in "processing"
Describe the bug
Some uploaded files are corrupted, and can't be viewed or edited. I experienced this while uploading a folder of images.
Trying to upload the folder of images as a compressed .zip file gets stuck in the "processing" state after upload.
Steps to reproduce
- Upload files via dialog or drag/drop.
Expected behavior
I would expect files to not be corrupted, and be available after upload.
Actual behavior
As described above, some files are corrupted on upload. I've tried deleting and re-uploading, but it's always the same files that become corrupted. This was tested with a folder of ~12 images.
Zipping the folder of images causes the program to hang, and the .zip is perpetually shown in the "processing" state.
This doesn't happen to all files– most seem to upload fine.
Setup
Docker compose AIO full stack, volumes are mounted via SMB/CIFS share.
.env file:
#IMPORTER=:web_extensions/importer.yml
# The importer needs additional config, see the following lines for more details.
## The docker image to be used for uppy companion.
COMPANION_IMAGE=
# Domain of Uppy Companion. Defaults to "companion.opencloud.test".
COMPANION_DOMAIN=
# Provider settings, see https://uppy.io/docs/companion/#provideroptions for reference.
# Empty by default, which disables providers.
COMPANION_ONEDRIVE_KEY=
COMPANION_ONEDRIVE_SECRET=
## Default Enabled Services ##
### Apache Tika Content Analysis Toolkit ###
# Tika (search) is enabled by default, comment if not required.
# Note: the leading colon is required to enable the service.
TIKA=:tika.yml
# Set the desired docker image tag or digest.
# Defaults to "latest"
TIKA_IMAGE=
### IMPORTANT Note for Online Office Apps ###
# To avoid app interlocking issues, you should select only one app to be active/configured.
# This is due the fact that there is currently no app interlocking for the same file and one
# has to wait for a lock release to open the file with another app.
### Collabora Settings ###
# Collabora web office is default enabled, comment if not required.
# Note: the leading colon is required to enable the service.
COLLABORA=:collabora.yml
# Domain of Collabora, where you can find the frontend.
# Defaults to "collabora.opencloud.test"
COLLABORA_DOMAIN=collabora.cloud.mydomain.com
# Domain of the wopiserver which handles OnlyOffice.
# Defaults to "wopiserver.opencloud.test"
WOPISERVER_DOMAIN=wopi.cloud.mydomain.com
# Admin user for Collabora.
# Defaults to "admin".
# Collabora Admin Panel URL:
# https://{COLLABORA_DOMAIN}/browser/dist/admin/admin.html
COLLABORA_ADMIN_USER=
# Admin password for Collabora.
# Defaults to "admin".
COLLABORA_ADMIN_PASSWORD=
# Set to true to enable SSL for Collabora Online. Default is true if not specified.
COLLABORA_SSL_ENABLE=false
# If you're on an internet-facing server, enable SSL verification for Collabora Online.
# Please comment out the following line:
COLLABORA_SSL_VERIFICATION=false
## Supplemental Configurations ##
# If you want to use supplemental configurations,
# you need to uncomment lines containing :path/file.yml
# and configure the service as required.
### Debugging - Monitoring ###
# Please see documentation at: https://opencloud.dev/opencloud/deployment/monitoring-tracing/
# Note: the leading colon is required to enable the service.
#MONITORING=:monitoring_tracing/monitoring.yml
### Virusscanner Settings ###
# IMPORTANT: If you enable antivirus, you also MUST configure the START_ADDITIONAL_SERVICES
# envvar in the OpenCloud Settings above by adding 'antivirus' to the list.
# Note: the leading colon is required to enable the service.
CLAMAV=:clamav.yml
# Image version of the ClamAV container.
# Defaults to "latest"
CLAMAV_DOCKER_TAG=
### OnlyOffice Settings ###
# Note: the leading colon is required to enable the service.
ONLYOFFICE=:onlyoffice.yml
# Domain for OnlyOffice. Defaults to "onlyoffice.opencloud.test".
ONLYOFFICE_DOMAIN=office.cloud.mydomain.com
# Domain for the wopiserver which handles OnlyOffice.
WOPISERVER_ONLYOFFICE_DOMAIN=wopi.cloud.mydomain.com
### Inbucket Settings ###
# Inbucket is a mail catcher tool for testing purposes.
# DO NOT use in Production.
# Note: the leading colon is required to enable the service.
#INBUCKET=:inbucket.yml
# email server (in this case inbucket acts as mail catcher).
# Domain for Inbucket. Defaults to "mail.opencloud.test".
INBUCKET_DOMAIN=
### Compose Configuration ###
# Path separator for supplemental compose files specified in COMPOSE_FILE.
COMPOSE_PATH_SEPARATOR=:
## IMPORTANT ##
# This MUST be the last line as it assembles the supplemental compose files to be used.
# ALL supplemental configs must be added here, whether commented or not.
# Each var must either be empty or contain :path/file.yml
COMPOSE_FILE=docker-compose.yml${OPENCLOUD:-}${TIKA:-}${DECOMPOSEDS3:-}${DECOMPOSEDS3_MINIO:-}${POSIX:-}${COLLABORA:-}${M>
Config:
GNU nano 7.2 opencloud.yml
opencloud:
image: ${OC_DOCKER_IMAGE:-opencloudeu/opencloud}:${OC_DOCKER_TAG:-latest}
# changelog: https://github.com/opencloud-eu/opencloud/tree/main/changelog
# release notes: https://docs.opencloud.eu/opencloud_release_notes.html
networks:
opencloud-net:
entrypoint:
- /bin/sh
# run opencloud init to initialize a configuration file with random secrets
# it will fail on subsequent runs, because the config file already exists
# therefore we ignore the error and then start the opencloud server
command: ["-c", "opencloud init || true; opencloud server"]
environment:
# enable services that are not started automatically
OC_ADD_RUN_SERVICES: ${START_ADDITIONAL_SERVICES}
OC_URL: https://${OC_DOMAIN:-cloud.opencloud.test}
OC_LOG_LEVEL: ${LOG_LEVEL:-info}
OC_LOG_COLOR: "${LOG_PRETTY:-false}"
OC_LOG_PRETTY: "${LOG_PRETTY:-false}"
# do not use SSL between Traefik and OpenCloud
PROXY_TLS: "false"
# make the REVA gateway accessible to the app drivers
GATEWAY_GRPC_ADDR: 0.0.0.0:9142
# INSECURE: needed if OpenCloud / Traefik is using self generated certificates
OC_INSECURE: "${INSECURE:-false}"
# basic auth (not recommended, but needed for eg. WebDav clients that do not support OpenID Connect)
PROXY_ENABLE_BASIC_AUTH: "${PROXY_ENABLE_BASIC_AUTH:-false}"
# admin user password
IDM_ADMIN_PASSWORD: "${ADMIN_PASSWORD:-admin}" # this overrides the admin password from the configuration file
# demo users
IDM_CREATE_DEMO_USERS: "${DEMO_USERS:-false}"
# email server (if configured)
NOTIFICATIONS_SMTP_HOST: "${SMTP_HOST}"
NOTIFICATIONS_SMTP_PORT: "${SMTP_PORT}"
NOTIFICATIONS_SMTP_SENDER: "${SMTP_SENDER:-OpenCloud notifications <notifications@${OC_DOMAIN:-cloud.opencloud.test>
NOTIFICATIONS_SMTP_USERNAME: "${SMTP_USERNAME}"
NOTIFICATIONS_SMTP_INSECURE: "${SMTP_INSECURE}"
# make the registry available to the app provider containers
MICRO_REGISTRY_ADDRESS: 127.0.0.1:9233
NATS_NATS_HOST: 0.0.0.0
NATS_NATS_PORT: 9233
PROXY_CSP_CONFIG_FILE_LOCATION: /etc/opencloud/csp.yaml
# these three vars are needed to the csp config file to include the web office apps and the importer
COLLABORA_DOMAIN: ${COLLABORA_DOMAIN:-collabora.opencloud.test}
ONLYOFFICE_DOMAIN: ${ONLYOFFICE_DOMAIN:-onlyoffice.opencloud.test}
COMPANION_DOMAIN: ${COMPANION_DOMAIN:-companion.opencloud.test}
# enable to allow using the banned passwords list
OC_PASSWORD_POLICY_BANNED_PASSWORDS_LIST: banned-password-list.txt
volumes:
- ./config/opencloud/app-registry.yaml:/etc/opencloud/app-registry.yaml
- ./config/opencloud/csp.yaml:/etc/opencloud/csp.yaml
- ./config/opencloud/banned-password-list.txt:/etc/opencloud/banned-password-list.txt
# configure the .env file to use own paths instead of docker internal volumes
- ${OC_CONFIG_DIR:-opencloud-config}:/etc/opencloud
- ${OC_DATA_DIR:-opencloud-data}:/var/lib/opencloud
labels:
- "traefik.enable=true"
- "traefik.http.routers.opencloud.entrypoints=https"
- "traefik.http.routers.opencloud.rule=Host(`${OC_DOMAIN:-cloud.opencloud.test}`)"
- "traefik.http.routers.opencloud.tls.certresolver=http"
- "traefik.http.routers.opencloud.service=opencloud"
- "traefik.http.services.opencloud.loadbalancer.server.port=9200"
logging:
driver: ${LOG_DRIVER:-local}
restart: always
volumes:
opencloud-config:
opencloud-data:
Additional context
See attached logs, and screenshots.
_opencloud_full-opencloud-1_logs.txt
That might be an issue of the smb/cifs mount. Nobody tried that before. OpenCloud needs extended attributes on the filesystem and caching might be an issue on this specific setup.
So what I am saying is: you are walking new ground with your setup, I did not expect it to work at all. Logs from the openCloud container would be interesting.
@aduffeck @dragotin
OpenCloud logs attached! What's odd is that many files have absolutely no issue, and some files do. If a file has an issue, it ALWAYS has an issue and reuploading it doesn't seem to work.
networkTimeoutError points into the direction that something low level is not responding.
networkTimeoutErrorpoints into the direction that something low level is not responding.
Actually in this specific case probably not. These are warning logged by https://github.com/tus/tusd/ likely because one of our http middleware lacks support for SetReadDeadline and SetWriteDeadline (See https://tus.github.io/tusd/advanced-topics/usage-package/#caveats for details). While we should address that, it should not cause imcomplete uploads.
networkTimeoutErrorpoints into the direction that something low level is not responding.Actually in this specific case probably not. These are warning logged by https://github.com/tus/tusd/ likely because one of our http middleware lacks support for
SetReadDeadlineandSetWriteDeadline(See https://tus.github.io/tusd/advanced-topics/usage-package/#caveats for details). While we should address that, it should not cause imcomplete uploads.
Thanks for this response! Just to clarify– when you say "it should not cause incomplete uploads", do you mean it's not the root cause of incomplete uploads? Or you think it is, and it should be fixed?
do you mean it's not the root cause of incomplete uploads?
I assume that it is not the root cause. AFAIR the message is currently be logged for every single tus upload. Even successful one.
Or you think it is, and it should be fixed?
It should be fixed anyway, but will likely not fix the problem you're seeing.
A couple of other datapoints for this issue:
I uploaded the same folder of files to another platform (CryptPad) with direct volume mounts, and had similar issues with the same files. Some of the photos appear to be much smaller in file size– most are ~10-15mb, while the three problem files were ~1.5mb. I'm not sure why they're so much smaller, but it seems to cause issues. This tells me that it is not an issue with the SMB mount, but could be an issue with processing certain types of picture files, since they work correctly on my local filesystem.
This does not explain, however, why there are issues with uploading a ZIP archive. I've tried uploading both an archive of this photo collection folder, and another of other files. It seems the larger ZIP files still become stuck in processing.
Unfortunately it seem that the log files you pasted are somewhat corrupted. Some lines seem to be cut and/or misplaced not sure what happened there.
I uploaded the same folder of files to another platform (CryptPad) with direct volume mounts, and had similar issues with the same files. Some of the photos appear to be much smaller in file size– most are ~10-15mb, while the three problem files were ~1.5mb. I'm not sure why they're so much smaller, but it seems to cause issues. This tells me that it is not an issue with the SMB mount
But if the same files cause issues with other platforms, this also seems to indicate that something else is fishy here which is likely unrelated to OpenCould. Unless I am completely misunderstanding you.
That might be an issue of the smb/cifs mount. Nobody tried that before. OpenCloud needs extended attributes on the filesystem and caching might be an issue on this specific setup.
I just found that some of the files uploaded to my instance have the same issue. I am using NFS (4.2) for the storage mount so it at least seems to occur on other networks filesystems too. I do not have logs as the upload happened a while back, but could try uploading it to another storage type if deemed necessary.
I have the same issue but with a non-network mounted volume, I uploaded 27 files and 2 files are still stuck at processing after a few days. (Also tried restarting/clearing the queue, but did not fix anything)
I also cannot upload new files or create new folders in this space anymore, other spaces in the same instance are working fine. When I try to create a folder the request times out and no log is displayed about it
