b2 sync: "ERROR:b2.bucket:error when uploading"

[bsilvern]

I'm using "b2 sync" to upload a series of 5GB files using 2 threads. For several different files, on multiple days, I've received "ERROR:b2.bucket:error when uploading" errors. As these are "large" files, there is no SHA-1 available on the target file.

Can I safely ignore these errors and assume that the sync command is smart enough to retry until success, or should I assume that these uploaded files are corrupted?

It would be helpful if b2 sync printed a status line at the completion of a sync operation that says something like:

• All files synced successfully.
• Errors occurred causing retries. All files synced successfully.
• Errors occurred, target file integrity is unknown.

Following is an example of a typical error message:

ERROR:b2.bucket:error when uploading, upload_url was https://pod-000-1040-05.backblaze.com/b2api/v1/b2_upload_part/4_[redacted].
Traceback (most recent call last):
  File "c:\program files\python\python36\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\program files\python\python36\lib\site-packages\urllib3\connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "c:\program files\python\python36\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1065, in _send_output
    self.send(chunk)
  File "c:\program files\python\python36\lib\http\client.py", line 986, in send
    self.sock.sendall(data)
  File "c:\program files\python\python36\lib\ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "c:\program files\python\python36\lib\ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "c:\program files\python\python36\lib\ssl.py", line 636, in write
    return self._sslobj.write(data)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python\python36\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "c:\program files\python\python36\lib\site-packages\urllib3\connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "c:\program files\python\python36\lib\site-packages\urllib3\util\retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "c:\program files\python\python36\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "c:\program files\python\python36\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\program files\python\python36\lib\site-packages\urllib3\connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "c:\program files\python\python36\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "c:\program files\python\python36\lib\http\client.py", line 1065, in _send_output
    self.send(chunk)
  File "c:\program files\python\python36\lib\http\client.py", line 986, in send
    self.sock.sendall(data)
  File "c:\program files\python\python36\lib\ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "c:\program files\python\python36\lib\ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "c:\program files\python\python36\lib\ssl.py", line 636, in write
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python\python36\lib\site-packages\b2\b2http.py", line 47, in _translate_errors
    response = fcn()
  File "c:\program files\python\python36\lib\site-packages\b2\b2http.py", line 255, in do_post
    response = self.session.post(url, headers=headers, data=data)
  File "c:\program files\python\python36\lib\site-packages\requests\sessions.py", line 560, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "c:\program files\python\python36\lib\site-packages\requests\sessions.py", line 513, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\program files\python\python36\lib\site-packages\requests\sessions.py", line 623, in send
    r = adapter.send(request, **kwargs)
  File "c:\program files\python\python36\lib\site-packages\requests\adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python\python36\lib\site-packages\b2\bucket.py", line 539, in _upload_part
    input_stream
  File "c:\program files\python\python36\lib\site-packages\b2\raw_api.py", line 467, in upload_part
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "c:\program files\python\python36\lib\site-packages\b2\b2http.py", line 259, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "c:\program files\python\python36\lib\site-packages\b2\b2http.py", line 103, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "c:\program files\python\python36\lib\site-packages\b2\b2http.py", line 76, in _translate_errors
    raise B2ConnectionError(str(e0))
b2.exception.B2ConnectionError: Connection error: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

[svonohr]

While there is no SHA1 checksum for the complete large file, each uploaded part of the file still uses a SHA1 checksum. If a transfer is interrupted while uploading a part, it tries reuploading this part again. From your posted log I can't really tell if the sync finished or was aborted because of the exception. If it finished successfully the uploaded files should be fine. If it aborted you can restart the sync and it will resume the unfinished large files (after checking that the SHA1 hash of all uploaded parts match the local file).

[bsilvern]

Thanks, svonohr. Just to be sure I re-ran the sync and it completed instantly with nothing printed. I also downloaded one of the files in question and it diff'd OK with the original. So it seems to work fine other than the intermittent comm failures and the resulting confusing python error stack it leaves behind.

[gwerbin]

I'm getting a similar intermittent error under Python 2.7. Same bug?

ERROR:b2.bucket:error when uploading, upload_url was <redacted>
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/b2/bucket.py", line 392, in _upload_small_file
    sha1_sum, file_info, input_stream
  File "/usr/local/lib/python2.7/site-packages/b2/raw_api.py", line 455, in upload_file
    return self.b2_http.post_content_return_json(upload_url, headers, data_stream)
  File "/usr/local/lib/python2.7/site-packages/b2/b2http.py", line 259, in post_content_return_json
    response = _translate_and_retry(do_post, try_count, post_params)
  File "/usr/local/lib/python2.7/site-packages/b2/b2http.py", line 103, in _translate_and_retry
    return _translate_errors(fcn, post_params)
  File "/usr/local/lib/python2.7/site-packages/b2/b2http.py", line 52, in _translate_errors
    int(error['status']), error['code'], error['message'], post_params
ServiceError: 500 internal_error Not enough members for write of <redacted>: read_only; class java.net.SocketException - Connection reset (http://member-08.vault:8180/api/put_shard?sguid=<redacted>&is_first=false&is_last=false&is_speed_test=false&upload_id=<redacted>&shard_size=2469425&offset=327680&reason=UPLOAD_B2)
upload Pictures/Photos/IMG_2266.MOV

[ppolewicz]

@gwerbin no: what you experienced was a temporary server-side error. The server is very careful and refuses to confirm a write as successful if it doesn't have enough spare erasure coding writes. It prefers safety over availability - which is what you want from a reliable cloud storage provider.

Try again in a short time - it should work then.

[mjurbanski-reef]

Please reopen if still occurs with the current version. CLI output should be clear whenever operation was a success or not.