python-egnyte
python-egnyte copied to clipboard
Published
20 hours ago •
egnyte
egnyte
bulk_upload times out or 502s
Without fail, running a client.bulk_upload() times out at 30 seconds, even on small files. If I invoke with a longer timeout, I get a 502 from the server.
Code used:
import egnyte
class verboselogger(egnyte.client.ProgressCallbacks):
force_newline = True
def creating_directory(self, cloud_folder):
sys.stderr.write('Creating %s\n' % (cloud_folder.path))
def upload_start(self, local_path, cloud_file, size):
sys.stderr.write('Starting Upload: %s (%d bytes)\n' % (cloud_file.path, size))
def upload_finish(self, cloud_file):
sys.stderr.write('Finished Upload: %s\n' % (cloud_file.path))
egnyte_path = '/Private/xxx/yyy/zzz'
client = egnyte.EgnyteClient({"domain": egnyte_domain, "access_token": egnyte_token})
pdf_dir = '/path/to/pdfs'
client.folder(egnyte_path).delete()
folder = client.folder(egnyte_path).create(ignore_if_exists=True)
client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())
Results:
Creating /Private/xxx/yyy/zzz
Creating /Private/xxx/yyy/zzz/
Starting Upload: /Private/xxx/yyy/zzz//_catalog.txt (1768 bytes)
Finished Upload: /Private/xxx/yyy/zzz//_catalog.txt
Starting Upload: /Private/xxx/yyy/zzz//index.pdf (1355115 bytes)
Traceback (most recent call last):
File "./test_create.py", line 183, in <module>
client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())
File "/home/user/.local/lib/python2.7/site-packages/egnyte/client.py", line 128, in bulk_upload
cloud_file.upload(fp, size, progress_callbacks.upload_progress)
File "/home/user/.local/lib/python2.7/site-packages/egnyte/resources.py", line 97, in upload
r = self._client.POST(url, data=chunk, headers={'Content-length': str(size)})
File "/home/user/.local/lib/python2.7/site-packages/egnyte/base.py", line 78, in POST
return self._retry(self._session.post, url, data=data, headers=headers, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/egnyte/base.py", line 57, in _retry
response = func(*args, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/user/.local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/user/.local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='xxxxxx.egnyte.com', port=443): Read timed out. (read timeout=30)
When invoked with the following:
client = egnyte.EgnyteClient({"domain": egnyte_domain, "access_token": egnyte_token, "timeout": "120"})
the results are as below:
Creating /Private/xxx/yyy/zzz
Creating /Private/xxx/yyy/zzz/
Starting Upload: /Private/xxx/yyy/zzz//_catalog.txt (1768 bytes)
Finished Upload: /Private/xxx/yyy/zzz//_catalog.txt
Starting Upload: /Private/xxx/yyy/zzz//index.pdf (1355115 bytes)
Traceback (most recent call last):
File "./test_create.py", line 183, in <module>
client.bulk_upload(['%s/' % (pdf_dir)], egnyte_path, progress_callbacks=verboselogger())
File "/home/user/.local/lib/python2.7/site-packages/egnyte/client.py", line 128, in bulk_upload
cloud_file.upload(fp, size, progress_callbacks.upload_progress)
File "/home/user/.local/lib/python2.7/site-packages/egnyte/resources.py", line 98, in upload
exc.default.check_response(r)
File "/home/user/.local/lib/python2.7/site-packages/egnyte/exc.py", line 168, in check_response
raise error_type(*errors)
egnyte.exc.RequestError: <RequestError: {url: 'https://xxxxxx.egnyte.com/pubapi/v1/fs-content/Private/xxx/yyy/zzz//index.pdf'}, {http response: '<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
'}, {http status: '502'}, {headers: '{'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'X-Egnyte-Request-Id': 'xxxxxxx:xxxxxxx:xxxx4_--_xxxxx|avl-www07.dc.egnyte.lan+https_l1_webui', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Cache-Control': 'no-cache', 'Content-Type': 'text/html'}'}>
Note that the failure does not always occur on the same file. Sometimes 1-2 files are uploaded fine. None of these are very large - 3MB at most.
The versions of the various requests modules are:
requests 2.22.0
requests-file 1.4.3
requests-toolbelt 0.8.0
This is on a CentOS 7 system and (for now) the script is using python 2.7.
