S3 upload still failes with OOM for 1.0.6

[ibayer]

The latest release still has the OOM bug when uploading to remote storage.

• nextcloud:23.0.2 with postgrest client added
• postgres:10.3

  > Health status: 0 correct, 363 missing and 0 faulty files
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00001-4J8cz2vkcCkabEn: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00002-KNuSXKImJx65wKm: PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 104857824 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error:  Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:311
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_6Qo...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_6Qo...', 'backups/2022032...')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
  thrown in /var/www/html/lib/private/Log.php on line 311

[Vrumf0ndel]

As a temporary workaround, you could increase the related setting in your php.ini file from 512M to a higher value, e.g. 768M. But be aware it's only a workaround and it does not attempt to fix the underlying problem. It can also cause your web server to run out of RAM.

; Maximum amount of memory a script may consume
; http://php.net/memory-limit
memory_limit=768M

[ibayer]

Would be interesting to see if this makes the app work or if other bugs pop up after this.

Knowing nothing about php I tried to find this file but :/srv/nextcloud# find . -name "php.ini" didn't point me to any php.ini file. I guess I can create this file somewhere but I'm not sure what would be the appropriate place with the official nextcloud docker container.

[Vrumf0ndel]

Just type php --ini and it will give you a lot of information. The information you're looking for is Loaded Configuration File: : e.g. Loaded Configuration File: /etc/php7/php.ini

Keep in mind that after restart of your docker container the changes are gone. So it's only for testing purposes.

[ibayer]

/etc/php7/php.ini doesn't exist and loaded is (none). :shrug:

Configuration File (php.ini) Path: /usr/local/etc/php
Loaded Configuration File:         (none)
Scan for additional .ini files in: /usr/local/etc/php/conf.d
Additional .ini files parsed:      /usr/local/etc/php/conf.d/docker-php-ext-apcu.ini,
/usr/local/etc/php/conf.d/docker-php-ext-bcmath.ini,
/usr/local/etc/php/conf.d/docker-php-ext-exif.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gd.ini,
/usr/local/etc/php/conf.d/docker-php-ext-gmp.ini,
/usr/local/etc/php/conf.d/docker-php-ext-imagick.ini,
/usr/local/etc/php/conf.d/docker-php-ext-intl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-ldap.ini,
/usr/local/etc/php/conf.d/docker-php-ext-memcached.ini,
/usr/local/etc/php/conf.d/docker-php-ext-opcache.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pcntl.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_mysql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-pdo_pgsql.ini,
/usr/local/etc/php/conf.d/docker-php-ext-redis.ini,
/usr/local/etc/php/conf.d/docker-php-ext-sodium.ini,
/usr/local/etc/php/conf.d/docker-php-ext-zip.ini,
/usr/local/etc/php/conf.d/nextcloud.ini,
/usr/local/etc/php/conf.d/opcache-recommended.ini

edit: I have put it here :/var/www/html# vim /usr/local/etc/php/conf.d/php.ini let see what happens.

[ibayer]

Seems to upload 1 more file before blowing up. :)

docker exec -it --user www-data nextcloud-f9ec7847-3b83-bf79-2049-578f2b137b3e php occ backup:point:upload 20220323114316-full-abXVdLfbkxKV5ZE
- uploading 20220323114316-full-abXVdLfbkxKV5ZE to external folders
 - checking external folder 19:backups
  > restoring point found
  > Health status: 2 correct, 360 missing and 1 faulty files
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00003-AySDZQMtYbrCAyc: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00004-Ohm7SQUAocykJVg: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00005-phENEUkPuNnLZ72: ok
  * Uploading data/data-e1a62938-57ed-4033-ab61-48b258866447/00006-YGr1UcnBOAp8w1w: PHP Fatal error:  Allowed memory size of 805306368 bytes exhausted (tried to allocate 104857704 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error:  Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:311
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_wlw...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_wlw...', 'backups/2022032...')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
  thrown in /var/www/html/lib/private/Log.php on line 311

[Vrumf0ndel]

I'm using different docker container, so it looks somehow different.

[Vrumf0ndel]

Looks like a bug in the upload routines where memory is not freed after single data chunk upload is finished.

[ibayer]

Jep, feels like a memory leak. Not something that a higher memory limit can fix. But I didn't check the code and know very little about php...

Anyway, thanks for the idea. :+1:

[varet80]

On the same train here! up to 8GB memory.. still fails

[domaximus]

set memory to -1 (unlimited) didn't fix it for me. triggering a manual upload uploads several files, but eventually the process stops with an event "Uploading Killed".

Any suggestions?

• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00010-BFx4FXwDTXFe7f2: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00011-8PyRfYtdV4qdlKy: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00012-Z86PNmryWJhS28j: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00013-o9si1auaT7XrwF2: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00014-bgVRndYRpr9m6dA: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00015-TeWpZ6ZGqkqO5uM: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00016-Ybzw1N63DLScwDc: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00017-hqiTqXfZrI7rIPH: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00018-RFT6XrIbyktKuYc: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00019-k4mUeJ13M9sif6U: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00020-L1rRdHLhOuzktjg: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00021-NKnFECss8JIvRNx: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00022-AWECeRYNX5M7COx: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00023-CvbcCU2TYtKBEEo: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00024-SDAoMYl3fXUR7t6: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00025-GAC8BaRjjUla4MU: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00026-XfPRZVGq5xULn3B: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00027-Ug3GGPmyIIqUqev: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00028-W1Vy2JbgYIyi9PI: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00029-7UCLU52pU4lWene: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00030-Jb5Z6RhyVFLZ2oC: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00031-Uwa3SZLJMdOmaGo: ok
• Uploading data/data-6bc4b442-d0d9-47d5-a979-85c2f74c3ce1/00032-AuKPsN8cuV9nnz3: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00001-vtBsU2EZqOe2X5G: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00002-hAAtq41oES5pKlq: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00003-xW98g1sMbyChn3s: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00004-anE7OAhOjc3FOiK: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00005-XBD1hHDH6bRrnAG: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00006-PsxDixhChzPkkie: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00007-nbQ9VT2olJdDszK: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00008-wg3v1wIZVKdmSkr: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00009-I4VkGzGViitl4QK: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00010-ZVUtqGzg4qE4LNl: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00011-QFbzv6z7dgxP5iZ: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00012-ZE4h96fMjX1fvEi: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00013-wrrsWBz15BnukpZ: ok
• Uploading data/data-046daf57-4fdb-496f-ab0c-e789e518ee66/00014-ALC8w1FP5nJ5b15: Killed

[fvillena]

Same problem on Backup 1.1.3

[SpaghettiCodeBooy]

Will this ever be fixed? This is really a deal-breaker for using NextCloud

[danepowell]

In the interest of consolidating discussion, would you mind closing this and upvoting #308?