django-dbbackup icon indicating copy to clipboard operation
django-dbbackup copied to clipboard

DB Backups to S3 over 5GB Fail

Open edmenendez opened this issue 4 years ago • 0 comments

Bug Report

Describe the bug

When trying to create a backup that's larger than 5GB it fails. I realize this is django-storages failing and it's because of a limitation of S3, but there should be a way of automatically handling this. This is supported by django-storages as 1.5.1 see this PR.

To Reproduce

Initiate a backup of a large DB.

Expected behavior

Expected it to create a save a backup file at S3.

Screenshots or reproduction

  File "manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "dbbackup/utils.py", line 99, in wrapper
    func(*args, **kwargs)
  File "dbbackup/management/commands/dbbackup.py", line 55, in handle
    self._save_new_backup(database)
  File "dbbackup/management/commands/dbbackup.py", line 81, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "dbbackup/management/commands/_base.py", line 83, in write_to_storage
    self.storage.write_file(file, path)
  File "dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "django/core/files/storage.py", line 52, in save
    return self._save(name, content)
  File "storages/backends/s3boto.py", line 433, in _save
    self._save_content(key, content, headers=headers)
  File "storages/backends/s3boto.py", line 444, in _save_content
    rewind=True, **kwargs)
  File "boto/s3/key.py", line 1309, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "boto/s3/key.py", line 762, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "boto/s3/key.py", line 963, in _send_file_internal
    query_args=query_args
  File "boto/s3/connection.py", line 671, in make_request
    retry_handler=retry_handler
  File "boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "boto/connection.py", line 1030, in _mexe
    raise ex
  File "boto/connection.py", line 940, in _mexe
    request.body, request.headers)
  File "boto/s3/key.py", line 856, in sender
    http_conn.send(chunk)
  File "http/client.py", line 993, in send
    self.sock.sendall(data)
  File "ssl.py", line 1034, in sendall
    v = self.send(byte_view[count:])
  File "ssl.py", line 1003, in send
    return self._sslobj.write(data)```

## Versions

### Django-dbbackup
- django-dbbackup==3.3.0
- django-storages==1.7.1
- boto==2.49.0

### External tools
- Python: 3.9.0
- Django: 2.2.20
- OS: Linux

edmenendez avatar Jul 13 '21 22:07 edmenendez