files_primary_s3 icon indicating copy to clipboard operation
files_primary_s3 copied to clipboard
verified publisher icon owncloud

Unable to complete upload of files >1GB even though all the blocks make it to the server in `uploads/...` and my s3 bucket

Open posita opened this issue 3 years ago • 6 comments

Migrated from owncloud/core#40100.

Errant behaviour

All uploads of files >1GB fail. More specifically, the attempt creates a server-side entry, but the hash does not match, which results in several destructive behaviors:

  1. The server-side entry is not downloadable;
  2. The desktop client will endlessly try to re-upload the file; and
  3. The desktop client with the files whose uploads have failed are starved from receiving further updates (e.g., other files uploaded from a separate client).

This has been a consistent behavior for me ever since first trying ownCloud almost two years ago. This report was from when I was running 10.9.1, but I have confirmed the behavior remains after upgrading to 10.10.0.3.

Server configuration

Operating system:

$ uname -srvmpio
Linux 4.14.117-grsec-grsec+ #1 SMP Fri May 10 17:15:47 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Web server: Dreamhost

Database: MySQL 8

PHP version: 7.4

.htaccess:

<IfModule mod_php5.c>
    php_value always_populate_raw_post_data -1
    php_value upload_max_filesize 4096M
    php_value post_max_size 4096M
    php_value memory_limit 4095M
    php_value mbstring.func_overload 0
    php_value default_charset 'UTF-8'
    php_value output_buffering 0
    <IfModule mod_env.c>
      SetEnv htaccessWorking true
    </IfModule>
</IfModule>

<IfModule mod_php7.c>
    php_value upload_max_filesize 4096M
    php_value post_max_size 4096M
    php_value memory_limit 4095M
    php_value mbstring.func_overload 0
    php_value default_charset 'UTF-8'
    php_value output_buffering 0
    <IfModule mod_env.c>
      SetEnv htaccessWorking true
    </IfModule>
</IfModule>

PHP INI:

upload_max_filesize=4096M
post_max_size=4096M
memory_limit=4095M
zend_extension=opcache.so

ownCloud version: (see ownCloud admin page)

$ php-7.4 ./occ -V
ownCloud 10.9.1

Updated from an older ownCloud or fresh install: fresh install

Where did you install ownCloud from: tarball

Signing status (ownCloud 9.0 and above):

No errors have been found.

The content of config/config.php:

$ php-7.4 ./occ config:list system
{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "…",
        "trusted_domains": [
            "localhost",
            "…"
        ],
        "datadirectory": "\/home\/…\/data",
        "tempdirectory": "\/home\/…\/data\/tmp",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbtype": "mysql",
        "version": "10.9.1.2",
        "dbhost": "…",
        "dbname": "…",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "logtimezone": "UTC",
        "session_lifetime": 3600,
        "allow_user_to_change_mail_address": "",
        "installed": true,
        "apps_paths": [
            {
                "path": "\/home\/…\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/home\/…\/apps-external",
                "url": "\/apps-external",
                "writable": true
            }
        ],
        "objectstore": {
            "class": "OCA\\Files_Primary_S3\\S3Storage",
            "arguments": {
                "bucket": "…",
                "options": {
                    "version": "2006-03-01",
                    "region": "us-west-1",
                    "credentials": {
                        "key": "***REMOVED SENSITIVE VALUE***",
                        "secret": "***REMOVED SENSITIVE VALUE***"
                    },
                    "endpoint": "http:\/\/s3.us-west-1.wasabisys.com\/",
                    "use_path_style_endpoint": true
                }
            }
        }
    }
}

List of activated apps:

$ php-7.4 ./occ app:list --enabled --minimal
Enabled:
  - activity 2.7.0
  - activity 2.7.0
  - calendar 1.6.4
  - comments 0.3.0
  - configreport 0.2.0
  - contacts 1.5.5
  - dav 0.7.0
  - federatedfilesharing 0.5.0
  - federation 0.1.0
  - files 1.5.2
  - files_external 0.8.0
  - files_mediaviewer 1.0.5
  - files_pdfviewer 1.0.1
  - files_primary_s3 1.2.0
  - files_sharing 0.14.0
  - files_texteditor 2.4.1
  - files_trashbin 0.9.1
  - files_versions 1.3.0
  - firstrunwizard 1.2.0
  - market 0.6.3
  - market 0.6.3
  - notifications 0.5.4
  - provisioning_api 0.5.0
  - systemtags 0.3.0
  - templateeditor 0.4.0
  - twofactor_totp 0.7.4
  - updatenotification 0.2.1

Are you using external storage, if yes which one: files_primary_s3

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Happens with both desktop client and web

Operating system: OS X Big Sur 11.6.5

Logs

Web server error log

N/A

ownCloud log (data/owncloud.log)

Attempt to download the inchoate server-side entry after a failed sync attempt:

{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…-…","level":4,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(321): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(67): Sabre\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}

Sync failure:

{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":4,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Server.php(345): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":4,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Server.php(345): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}

Client side error once errant server-side entry is deleted and re-sync attempt is made:

Issues         ,File                                                            ,Folder                        ,Size  ,Account             ,Time                ,Status              ,
Server replied "500 Internal Server Error" to "MOVE https://…/remote.php/dav/uploads/…/…/.file",…                                ,ownCloud                      ,1 GB  ,…,2022-05-24T10:47:03.000,Error               ,

posita avatar Aug 28 '22 20:08 posita

I have verified the values for upload_max_filesize, post_max_size, and memory_limit:

image

posita avatar Aug 28 '22 20:08 posita

For whatever it is worth: I have tested ownCloud server 10.11.0 with files_primary_s3 1.4.0 sitting on a scality/cloudserver backend:

  • Upload up to 21GB works
  • (then I got disk full)

jnweiger avatar Nov 02 '22 14:11 jnweiger

UPDATE: I am still experiencing this issue after upgrading to 10.11.0 and can reproduce it with both Wasabi and DreamObjects as S3 back-ends. This is from my latest attempt to sync a 1GB file from the web interface:

{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"Error executing \"GetObject\" on \"https:\/\/objects-us-east-1.dream.io\/[redacted]\/urn%3Aoid%3A12345\"; AWS HTTP error: Client error: `GET https:\/\/objects-us-east-1.dream.io\/[redacted]\/urn%3Aoid%3A12345` resulted in a `404 Not Found` response NotFound (client): 404 Not Found (Request-ID: tx[redacted]-us-east-1-iad1) - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>NoSuchKey<\/Code><BucketName>[redacted]<\/BucketName><RequestId>tx[redacted]-us-east-1-iad1<\/RequestId><HostId>[redacted]-us-east-1-iad1-us-east-1<\/HostId><\/Error> at \/[path]\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"fopen(s3:\/\/[redacted]\/urn:oid:12345): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/[path]\/apps-external\/files_primary_s3\/lib\/s3storage.php#267"}
{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"fread() expects parameter 1 to be resource, bool given at \/[path]\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#932"}
{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"fclose() expects parameter 1 to be resource, bool given at \/[path]\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#933"}
{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"Error executing \"GetObject\" on \"https:\/\/objects-us-east-1.dream.io\/[redacted]\/urn%3Aoid%3A12345\"; AWS HTTP error: Client error: `GET https:\/\/objects-us-east-1.dream.io\/[redacted]\/urn%3Aoid%3A12345` resulted in a `404 Not Found` response NotFound (client): 404 Not Found (Request-ID: tx[redacted]-us-east-1-iad1) - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>NoSuchKey<\/Code><BucketName>[redacted]<\/BucketName><RequestId>tx[redacted]-us-east-1-iad1<\/RequestId><HostId>[redacted]-us-east-1-iad1-us-east-1<\/HostId><\/Error> at \/[path]\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"[redacted]","level":3,"time":"2022-11-14T22:47:56+00:00","remoteAddr":"[redacted]","user":"[user]","app":"PHP","method":"MOVE","url":"\/remote.php\/dav\/uploads\/[user]\/3300640080\/.file","message":"fopen(s3:\/\/[redacted]\/urn:oid:12345): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/[path]\/apps-external\/files_primary_s3\/lib\/s3storage.php#267"}

And after an attempt via sync:

Issues         ,File                                                            ,Folder                        ,Size  ,Account             ,Time                ,Status              ,
Server replied "423 Locked" to "MOVE https://[redacted]/remote.php/dav/uploads/[user]/414185728/.file" ("[filename]" is locked) (skipped due to earlier error, trying again in 2 minute(s)),[filename]                                                    ,ownCloud                      ,      ,[user]@[redacted],2022-11-14T18:00:35.101,Blacklisted         ,

posita avatar Nov 14 '22 23:11 posita

Thanks @jnweiger, did you try with a single 21GB file? Is this the S3 backend you're using?

posita avatar Nov 16 '22 13:11 posita

Okay, digging into this a bit more, I am able to upload all the blocks successfully, at least via the web interface. I believe this is also true via the Desktop client. Either way, I am able to confirm that the blocks are uploaded successfully to, e.g., uploads/web-file-upload-c337...3e55-166...959. In fact, I can retrieve them from there via davs://<host>/remote.php/dav/uploads/<user>/web-file-upload-c337...3e55-166...959, and reassemble them to get a byte-for-byte identical copy to the original file. I can also confirm that the blocks are making it to S3 by querying oc_filecache and looking for those blocks:

SELECT fc.fileid
FROM oc_filecache fc
JOIN oc_storages s ON s.numeric_id = fc.storage
WHERE fc.path LIKE 'uploads/web-file-upload-c337...3e55-166...959/%'
AND s.id = 'object::user:<user>' ORDER BY fileid ASC

#!/usr/bin/env bash
set -eux -o pipefail
for fileid in <results-from-above-sql-query> ; do
  aws s3 \
    --profile dreamobjects \
    --endpoint-url 'https://objects-us-east-1.dream.io/' \
    cp "s3://<redacted>/urn:oid:${fileid}" ".../s3-blocks/${fileid}"
done
( IFS=$'\n' ; for f in $( \ls .../s3-blocks | sort -n ) ; do cat ".../s3-blocks/${f}" ; done ) | sha256sum

:point_up: That will give me the same checksum as ( IFS=$'\n' ; for f in $( \ls .../web-file-upload-c337...3e55-166...959 | sort -n ) ; do cat ".../web-file-upload-c337...3e55-166...959/${f}" ; done ) | sha256sum, which will give me the same checksum as sha256sum <original-file>.

So whatever is failing is failing after the initial (chunked) upload.

posita avatar Nov 21 '22 15:11 posita

Okay, here's the rub. I can create a one-byte placeholder file at the desired path (e.g., folder/BIGFILE). I can find the fileid as follows:

SELECT fc.fileid
FROM oc_filecache fc
JOIN oc_storages s ON s.numeric_id = fc.storage
WHERE fc.path = 'files/folder/BIGFILE'
AND s.id = 'object::user:<user>' ORDER BY fileid ASC

Let's say that gives me fileid 1234. Now I can go into maintenance mode and copy the actual big file to that S3 bucket:

aws s3 ... cp .../BIGFILE "s3://<redacted>/urn:oid:1234"

I can go back and hand-edit the oc_filecache entry to update the size, mtime, storage_mtime, and checksum.

storage_mtime:

date --date "$( aws s3 ... ls "s3://<redacted>/urn:oid:1234" | awk '{ print $1, $2; }' )" +%s

checksum:

o=.../BIGFILE
printf 'SHA1:%s MD5:%s ADLER32:%08x\n' \
  "$( sha1sum "${o}" | awk '{ print $1; }' )" \
  "$( md5sum "${o}" | awk '{ print $1; }' )" \
  "$( python3 -c 'import io, sys, zlib ; print(zlib.adler32(open(sys.argv[1], "rb").read()))' "${o}" )"

I just don't know how to compute a new etag value, so I have to do something else (like a move or delete/restore) to trigger a Desktop client refresh if I forgot to shut it down before it grabbed the one-byte placeholder file. But, once all those operations are complete, I can turn maintenance mode off and there's folder/BIGFILE in all its glory, downloadable via the web interface and the Desktop client, and verifiable as the same file that I started with.

Obviously that's not a viable workflow, but I'm hoping it helps narrow down where the problem lives.

posita avatar Nov 21 '22 16:11 posita