OOM while uploading chunks

[Maffsie]

Hi!

I'm trying to set up backups on my Nextcloud 23.0.10 instance using this app - version 1.0.4. Creating an initial full backup manually with ./occ backup:point:create succeeds, as does ./occ backup:point:pack [ID], however once I run ./occ backup:point:upload, it begins uploading all chunks for the backup, and then eventually it runs out of memory, and I have to unlock the backup and then restart the upload. My PHP memory limit is set to 4GB and I'm running this inside Docker, and my external storage is a DigitalOcean Spaces bucket.

Logs:

www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16
- uploading DATE-full-ID to external folders
 - checking external folder 16:PATH
  > restoring point not found
  * Creating Restoring Point on external folder: ok
  > restoring point created
  * initiating health check
  > Health status: 0 correct, 541 missing and 0 faulty files
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00001-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00002-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00003-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00004-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00005-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00006-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00007-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00008-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00009-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00010-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00011-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00012-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00013-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00014-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00015-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00016-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00017-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00018-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00019-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00020-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00021-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00022-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00023-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00024-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00025-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00026-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00027-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00028-...: ok
  * Uploading data/data-5008c857-0c7f-49bc-a318-.../00029-...: PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 139810256 bytes) in /var/www/html/custom_apps/backup/lib/Service/ExternalFolderService.php on line 262
www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16

In MetadataService.php line 155:

  point is locked

[...snip...]

www-data@8865cf96ff87:~/html$ ./occ backup:point:unlock DATE-full-ID
www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16
- uploading DATE-full-ID to external folders
 - checking external folder 16:PATH
  > restoring point found
  > Health status: 83 correct, 457 missing and 1 faulty files
  * Uploading data/data-08152021-9fab-492b-8735-.../00004-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00005-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00006-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00007-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00008-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00009-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00010-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00011-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00012-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00013-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00014-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00015-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00016-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00017-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00018-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00019-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00020-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00021-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00022-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00023-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00024-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00025-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00026-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00027-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00028-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00029-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00030-...: ok
  * Uploading data/data-08152021-9fab-492b-8735-.../00031-...: PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 104857648 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error:  Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:316
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_EWe...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_EWe...', 'PATH')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
  thrown in /var/www/html/lib/private/Log.php on line 316
www-data@8865cf96ff87:~/html$

[danepowell]

In the interest of consolidating discussion, would you mind closing this and upvoting #308?