openeew-python icon indicating copy to clipboard operation
openeew-python copied to clipboard

bug with get_filtered_records function

Open mgrobaker opened this issue 5 years ago • 3 comments

I am not able to download data using the python library. It could be an issue on my end, but I don't think so

When I follow instructions here, I get this error

Traceback (most recent call last):
  File "test_openeew.py", line 18, in <module>
    eew_data = eew_aws_client.get_filtered_records(start_time, end_time)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/openeew/data/aws.py", line 350, in get_filtered_records
    key_records = loop.run_until_complete(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/openeew/data/aws.py", line 305, in _download_keys
    key_records = await asyncio.gather(*coros)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/openeew/data/aws.py", line 276, in _get_records_from_key
    await async_s3_client.download_fileobj(
AttributeError: 'ClientCreatorContext' object has no attribute 'download_fileobj'
sys:1: RuntimeWarning: coroutine 'AioSession._create_client' was never awaited

mgrobaker avatar Oct 19 '20 15:10 mgrobaker

I'm having the exact same issue.

Trying to execute code as written in https://openeew.com/docs/analyze-historic gives

RuntimeError                              Traceback (most recent call last)
<ipython-input-10-5ac17f81d3ef> in <module>
----> 1 records = data_client.get_filtered_records(start_date_utc, end_date_utc)

D:\Applications\Anaconda3\lib\site-packages\openeew\data\aws.py in get_filtered_records(self, start_date_utc, end_date_utc, device_ids)
    348         loop = asyncio.get_event_loop()
    349 
--> 350         key_records = loop.run_until_complete(
    351                 self._download_keys(keys_to_download)
    352                 )

D:\Applications\Anaconda3\lib\asyncio\base_events.py in run_until_complete(self, future)
    590         """
    591         self._check_closed()
--> 592         self._check_running()
    593 
    594         new_task = not futures.isfuture(future)

D:\Applications\Anaconda3\lib\asyncio\base_events.py in _check_running(self)
    550     def _check_running(self):
    551         if self.is_running():
--> 552             raise RuntimeError('This event loop is already running')
    553         if events._get_running_loop() is not None:
    554             raise RuntimeError(

RuntimeError: This event loop is already running

Executing the code with the nested asynchio fix demonstrated in this repo's jupyter notebook

# Allow nested asyncio event loop
# See https://github.com/erdewit/nest_asyncio
import nest_asyncio
nest_asyncio.apply()

then produces the same error as @mgrobaker .

pdornian avatar Nov 08 '20 23:11 pdornian

Thoughts on this @andygrillo?

krook avatar Nov 11 '20 16:11 krook

Thanks for reporting this error. It looks like it's caused by a breaking change in aiobotocore to do with how clients are created: https://github.com/aio-libs/aiobotocore/blob/master/CHANGES.rst#100-2020-03-31.

As mentioned in https://github.com/terrycain/aioboto3#async-aws-sdk-for-python, clients must now be created as async context managers. As it happens, this change has already been made as part of the unreleased https://github.com/openeew/openeew-python/pull/10. Until a new version of the library has been released, using the current master branch should work.

mallmanj avatar Mar 21 '21 20:03 mallmanj