core Unable to upload files >1GB with `files_primary

Unable to upload files >1GB with `files_primary_s3`

Open posita opened this issue 2 years ago • 1 comments

Errant behaviour

All uploads of files >1GB fail. More specifically, the attempt creates a server-side entry, but the hash does not match, which results in several destructive behaviors:

The server-side entry is not downloadable;
The desktop client will endlessly try to re-upload the file; and
The desktop client with the files whose uploads have failed are starved from receiving further updates (e.g., other files uploaded from a separate client).

Server configuration

Operating system:

$ uname -srvmpio
Linux 4.14.117-grsec-grsec+ #1 SMP Fri May 10 17:15:47 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux

Web server: Dreamhost

Database: MySQL 8

PHP version: 7.4

.htaccess:

<IfModule mod_php5.c>
    php_value always_populate_raw_post_data -1
    php_value upload_max_filesize 4096M
    php_value post_max_size 4096M
    php_value memory_limit 4096M
    php_value mbstring.func_overload 0
    php_value default_charset 'UTF-8'
    php_value output_buffering 0
    <IfModule mod_env.c>
      SetEnv htaccessWorking true
    </IfModule>
</IfModule>

<IfModule mod_php7.c>
    php_value upload_max_filesize 4096M
    php_value post_max_size 4096M
    php_value memory_limit 4096M
    php_value mbstring.func_overload 0
    php_value default_charset 'UTF-8'
    php_value output_buffering 0
    <IfModule mod_env.c>
      SetEnv htaccessWorking true
    </IfModule>
</IfModule>

PHP INI:

upload_max_filesize=4096M
post_max_size=4096M
memory_limit=4096M
zend_extension=opcache.so

ownCloud version: (see ownCloud admin page)

$ php-7.4 ./occ -V
ownCloud 10.9.1

Updated from an older ownCloud or fresh install: fresh install

Where did you install ownCloud from: tarball

Signing status (ownCloud 9.0 and above):

No errors have been found.

The content of config/config.php:

$ php-7.4 ./occ config:list system
{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "…",
        "trusted_domains": [
            "localhost",
            "…"
        ],
        "datadirectory": "\/home\/…\/data",
        "tempdirectory": "\/home\/…\/data\/tmp",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbtype": "mysql",
        "version": "10.9.1.2",
        "dbhost": "…",
        "dbname": "…",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "logtimezone": "UTC",
        "session_lifetime": 3600,
        "allow_user_to_change_mail_address": "",
        "installed": true,
        "apps_paths": [
            {
                "path": "\/home\/…\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/home\/…\/apps-external",
                "url": "\/apps-external",
                "writable": true
            }
        ],
        "objectstore": {
            "class": "OCA\\Files_Primary_S3\\S3Storage",
            "arguments": {
                "bucket": "…",
                "options": {
                    "version": "2006-03-01",
                    "region": "us-west-1",
                    "credentials": {
                        "key": "***REMOVED SENSITIVE VALUE***",
                        "secret": "***REMOVED SENSITIVE VALUE***"
                    },
                    "endpoint": "http:\/\/s3.us-west-1.wasabisys.com\/",
                    "use_path_style_endpoint": true
                }
            }
        }
    }
}

List of activated apps:

$ php-7.4 ./occ app:list --enabled --minimal
Enabled:
  - activity 2.7.0
  - activity 2.7.0
  - calendar 1.6.4
  - comments 0.3.0
  - configreport 0.2.0
  - contacts 1.5.5
  - dav 0.7.0
  - federatedfilesharing 0.5.0
  - federation 0.1.0
  - files 1.5.2
  - files_external 0.8.0
  - files_mediaviewer 1.0.5
  - files_pdfviewer 1.0.1
  - files_primary_s3 1.2.0
  - files_sharing 0.14.0
  - files_texteditor 2.4.1
  - files_trashbin 0.9.1
  - files_versions 1.3.0
  - firstrunwizard 1.2.0
  - market 0.6.3
  - market 0.6.3
  - notifications 0.5.4
  - provisioning_api 0.5.0
  - systemtags 0.3.0
  - templateeditor 0.4.0
  - twofactor_totp 0.7.4
  - updatenotification 0.2.1

Are you using external storage, if yes which one: files_primary_s3

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Happens with both desktop client and web

Operating system: OS X Big Sur 11.6.5

Logs

Web server error log

N/A

ownCloud log (data/owncloud.log)

Attempt to download the inchoate server-side entry after a failed sync attempt:

{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…-…","level":3,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…-…","level":4,"time":"2022-05-24T14:43:56+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/…?downloadStartSecret=…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(321): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v1\\\/webdav.php(67): Sabre\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}

Sync failure:

{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":4,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Server.php(345): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fread() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#927"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fclose() expects parameter 1 to be resource, bool given at \/home\/…\/lib\/private\/Files\/Storage\/Wrapper\/Encryption.php#928"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Error executing \"GetObject\" on \"http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A…\"; AWS HTTP error: Client error response [url] http:\/\/s3.us-west-1.wasabisys.com\/…\/urn%3Aoid%3A… [status code] 404 [reason phrase] Not Found NotFound (client): 404 Not Found (Request-ID: …) - <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchKey<\/Code><Message>The specified key does not exist.<\/Message><Key>urn:oid:…<\/Key><RequestId>…<\/RequestId><HostId>…<\/HostId><\/Error> at \/home\/…\/apps-external\/files_primary_s3\/lib\/streamwrapper.php#721"}
{"reqId":"…","level":3,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"PHP","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"fopen(s3:\/\/…\/urn:oid:…): failed to open stream: &quot;OCA\\Files_Primary_S3\\StreamWrapper::stream_open&quot; call failed at \/home\/…\/apps-external\/files_primary_s3\/lib\/s3storage.php#184"}
{"reqId":"…","level":4,"time":"2022-05-24T15:00:00+00:00","remoteAddr":"…","user":"…","app":"webdav","method":"GET","url":"\/remote.php\/dav\/files\/…","message":"Exception: HTTP\/1.1 503 Could not open file: {\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(85): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#1 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/WildcardEmitterTrait.php(89): Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#2 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(472): Sabre\\\\DAV\\\\Server->emit('method:GET', Array)\\n#3 \\\/home\\\/…\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(253): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#4 \\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Server.php(345): Sabre\\\\DAV\\\\Server->start()\\n#5 \\\/home\\\/…\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#6 \\\/home\\\/…\\\/remote.php(165): require_once('\\\/home\\\/...')\\n#7 {main}\",\"File\":\"\\\/home\\\/…\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":404}"}

Client side error once errant server-side entry is deleted and re-sync attempt is made:

Issues         ,File                                                            ,Folder                        ,Size  ,Account             ,Time                ,Status              ,
Server replied "500 Internal Server Error" to "MOVE https://…/remote.php/dav/uploads/…/…/.file",…                                ,ownCloud                      ,1 GB  ,…,2022-05-24T10:47:03.000,Error               ,

May 24 '22 15:05 posita

FYI, this has been a consistent behavior for me ever since first trying ownCloud over a year ago.

Jun 05 '22 22:06 posita

Friendly ping. I am still observing this behavior. Any ideas or advice?

Aug 25 '22 01:08 posita

https://github.com/owncloud/files_primary_s3/issues could better fit? Tryed upgradation ownCloud 10.10.0?

Aug 27 '22 14:08 ho4ho

I am still experiencing the issue with 10.10.0.3. Refiled as recommended as owncloud/files_primary_s3#601.

Aug 28 '22 20:08 posita

core core copied to clipboard

Unable to upload files >1GB with `files_primary_s3`

Errant behaviour

Server configuration

Client configuration

Logs

Web server error log

ownCloud log (data/owncloud.log)

core
core copied to clipboard