S3 to DC tool fails by hanging indefinitely

[alexgleith]

It's hard to reproduce, but if you install odc-apps-dc-tools without aiohttp==3.6.2, for example, the tool will just hang.

[Kirill888]

Are you using "allow pre-release installs" flags? We should only install our libs with those flags. I think for aio libs it's best to pin them even if this is not the cause of this problem.

[alexgleith]

This is the requirements used:

--pre
--extra-index-url https://packages.dea.ga.gov.au/
datacube[performance,s3]==1.8.3
aiobotocore[boto3,awscli]==1.1.1
odc-apps-dc-tools

Adding in pinned aiohttp makes it work.

[Kirill888]

same as #76, don't use --pre globally, right now it's aiohttp that is breaking, but next some other thing will, --pre should only apply to odc-apps-dc-tools really, and you can avoid using --pre altogether by specifying >={dev_version} on odc-apps-dc-tools.

[alexgleith]

Please create a PR on the requirements files in here: https://github.com/opendatacube/datacube-docker

But it is patching the symptom, and doesn't resolve that the tool should fail faster if it's broken and not hang indefinitely.

[Kirill888]

have you debugged which library it is hanging in?

Nothing I can do if your environment has incompatible versions of unreleased third party software that happen to misbehave when used that way.

[woodcockr]

I reported this previously. It's not a library issue. Something to do with asyncio race condition when something fails (something related to network, outside the core logic flow anyway) was as far as I could theorize but my skills with aio are extremely limited and I couldn't figure out how to debug it further. Uncaught exception maybe? its quite painful to reproduce but it happens a fair bit, unfortunately you can run the same thing twice and have it not happen for one of them. quite frustrating for debugging.

[Kirill888]

I have not seen it happen when using non-alpha versions of aiohttp, aiobotocore installed with compatible botocore