Broken Connection for get_asset

[scleronomic]

Describe the Bug

I can only very inconsistently download assets. The larger the assets are, the more often it fails. Smaller then 0.5Mb works fine most of the time, larger than 5Mb fails 95% of the time. Right now I am sometimes able to 'solve' this problem by making a loop around the .get_asset call and repeat it until it succeeds.

Manually downloading the file form the website also fails most of the times with the download just timing out.

Expected behaviour

Expected behaviour is that the downloads don't time out. I don't know if there are any size limits or frequencies of API calls of that I am not aware of. If this is the case, then

Where is the issue?

• [X ] Comet Python SDK
• [X] Comet UI
• [ ] Third Party Integrations (Huggingface, TensorboardX, Pytorch Lighting etc)

To Reproduce

The code I use to download a given asset is:

api = comet_ml.API(api_key=COMET_API_KEY)
exp = api.get(workspace=workspace, project_name=project_name, experiment=experiment)
m_bin = experiment.get_asset(asset_id, return_type='binary')

Stack Trace

If possible please include the full stack trace of your issue here

Traceback (most recent call last):
  File "/Users/jote/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/222.3739.56/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/api.py", line 1365, in get_asset
    results = self._api._client.get_experiment_asset(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 2293, in get_experiment_asset
    response = self.get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 4301, in get_from_endpoint
    retval = super(RestApiClientWithCache, self).get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 1901, in get_from_endpoint
    retval = response.content
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
m_bin = experiment.get_asset(asset_id, return_type='binary', stream=True)
Traceback (most recent call last):
  File "/Users/jote/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/222.3739.56/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/api.py", line 1365, in get_asset
    results = self._api._client.get_experiment_asset(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 2293, in get_experiment_asset
    response = self.get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 4301, in get_from_endpoint
    retval = super(RestApiClientWithCache, self).get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 1901, in get_from_endpoint
    retval = response.content
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
m_bin = experiment.get_asset(asset_id, return_type='binary', stream=True)
Traceback (most recent call last):
  File "/Users/jote/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/222.3739.56/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/api.py", line 1365, in get_asset
    results = self._api._client.get_experiment_asset(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 2293, in get_experiment_asset
    response = self.get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 4301, in get_from_endpoint
    retval = super(RestApiClientWithCache, self).get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 1901, in get_from_endpoint
    retval = response.content
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
m_bin = experiment.get_asset(asset_id, return_type='binary', stream=True)
Traceback (most recent call last):
  File "/Users/jote/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/222.3739.56/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/api.py", line 1365, in get_asset
    results = self._api._client.get_experiment_asset(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 2293, in get_experiment_asset
    response = self.get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 4301, in get_from_endpoint
    retval = super(RestApiClientWithCache, self).get_from_endpoint(
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/comet_ml/connection.py", line 1901, in get_from_endpoint
    retval = response.content
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/jote/miniconda3/envs/py38/lib/python3.8/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[DN6]

Hello @scleronomic. Thanks for flagging this. Would it be possible to share the Experiment that you're trying to download the asset from? (can it be made public?)

Also, if you could run your snippet with Comet Debug mode enabled . To run Comet in debug mode, set the following environment variables. If you can share the resulting comet.log file with us, it will help us understand what might be happening a bit better.

export COMET_LOGGING_FILE=/tmp/comet.log
export COMET_LOGGING_FILE_LEVEL=debug

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.