dask_cloudprovider.utils.timeout.TimeoutException: Failed to find scheduler ip address after 120 seconds

[sergii-ivakhno-kidsloop]

I am getting timeout after launching Fargate cluster dask_cloudprovider.utils.timeout.TimeoutException: Failed to find scheduler ip address after 120 seconds. I know this error has been reported before, but in my case it doesn't seem to be related to timeout duration as I get the same error after waiting for 5min

Looking into the source code it seems that the issue is due to parsing and getting IP from logs at https://github.com/dask/dask-cloudprovider/blob/main/dask_cloudprovider/aws/ecs.py#L209 It tries to access events within logs but it is empty when I print it at this line https://github.com/dask/dask-cloudprovider/blob/main/dask_cloudprovider/aws/ecs.py#L379

{'events': [], 'nextForwardToken': 'f/36723491871038569079854395880864976970354492776673968128/s', 'nextBackwardToken': 'b/36723331305673139659367776823693413902151649717678768128/s', 'ResponseMetadata': {'RequestId': 'a4af94b5-b125-44c5-8b22-8455cf0c07e4', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'a4af94b5-b125-44c5-8b22-8455cf0c07e4', 'content-type': 'application/x-amz-json-1.1', 'content-length': '178', 'date': 'Tue, 08 Mar 2022 09:18:47 GMT'}, 'RetryAttempts': 0}}

What happened:

TimeoutException

What you expected to happen:

No failure

Minimal Complete Verifiable Example:

cluster = FargateCluster(
        image="prefecthq/prefect:latest",
        execution_role_arn="arn:aws:iam::**:role/dusk-cluster",
        task_role_policies = ["arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"],
        platform_version='1.3.0',
        n_workers=3,
        scheduler_cpu=256,
        scheduler_mem=512,
        worker_cpu=256,
        worker_mem=512,
        scheduler_timeout="15 minutes",
        find_address_timeout=120
    )

Anything else we need to know?:

Environment:

• Dask version: dask-cloudprovider-2022.1.0, dask-2022.2.1
• Python version: Python 3.8.12
• Operating System: Ubuntu 18.04.6 LTS
• Install method (conda, pip, source): Poetry

Cluster Dump State: