django-dbbackup
django-dbbackup copied to clipboard
DB Backups to S3 over 5GB Fail
Bug Report
Describe the bug
When trying to create a backup that's larger than 5GB it fails. I realize this is django-storages failing and it's because of a limitation of S3, but there should be a way of automatically handling this. This is supported by django-storages as 1.5.1 see this PR.
To Reproduce
Initiate a backup of a large DB.
Expected behavior
Expected it to create a save a backup file at S3.
Screenshots or reproduction
File "manage.py", line 17, in <module>
execute_from_command_line(sys.argv)
File "django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "dbbackup/utils.py", line 99, in wrapper
func(*args, **kwargs)
File "dbbackup/management/commands/dbbackup.py", line 55, in handle
self._save_new_backup(database)
File "dbbackup/management/commands/dbbackup.py", line 81, in _save_new_backup
self.write_to_storage(outputfile, filename)
File "dbbackup/management/commands/_base.py", line 83, in write_to_storage
self.storage.write_file(file, path)
File "dbbackup/storage.py", line 82, in write_file
self.storage.save(name=filename, content=filehandle)
File "django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "storages/backends/s3boto.py", line 433, in _save
self._save_content(key, content, headers=headers)
File "storages/backends/s3boto.py", line 444, in _save_content
rewind=True, **kwargs)
File "boto/s3/key.py", line 1309, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "boto/s3/key.py", line 762, in send_file
chunked_transfer=chunked_transfer, size=size)
File "boto/s3/key.py", line 963, in _send_file_internal
query_args=query_args
File "boto/s3/connection.py", line 671, in make_request
retry_handler=retry_handler
File "boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "boto/connection.py", line 1030, in _mexe
raise ex
File "boto/connection.py", line 940, in _mexe
request.body, request.headers)
File "boto/s3/key.py", line 856, in sender
http_conn.send(chunk)
File "http/client.py", line 993, in send
self.sock.sendall(data)
File "ssl.py", line 1034, in sendall
v = self.send(byte_view[count:])
File "ssl.py", line 1003, in send
return self._sslobj.write(data)```
## Versions
### Django-dbbackup
- django-dbbackup==3.3.0
- django-storages==1.7.1
- boto==2.49.0
### External tools
- Python: 3.9.0
- Django: 2.2.20
- OS: Linux