prefect Zombie Tasks Runs caused by increasing concurrency limit during task run

Zombie Tasks Runs caused by increasing concurrency limit during task run

Open j-tr opened this issue 2 years ago • 3 comments

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

when increasing the task concurrency limit while a flow is executing (this would be very useful when having long running flows and a dynamically scaleable db), the number of "Active Tasks Runs" doesn't return to zero after all flows are completed. instead a certain number of tasks is still listed as active by the prefect concurrency-limit inspect command but is in fact in status completed.

Reproduction

import time
from prefect import flow, task, tags, context, get_run_logger
from prefect.utilities.asyncutils import sync_compatible


tag = "test-limit"


@sync_compatible
async def create_limit(n):
    # set concurrency limit
    await context.get_run_context().client.create_concurrency_limit(tag=tag, concurrency_limit=n)


@sync_compatible
async def get_active_tasks():
    # list all "active" tasks for the tag
    return (await context.get_run_context().client.read_concurrency_limit_by_tag(tag=tag)).active_slots


@sync_compatible
async def get_running_tasks():
    # list all tasks ids in state "Running"
    running_tasks = []
    for future in context.get_run_context().task_run_futures:
        if (await future._get_state()).name == "Running":
            running_tasks.append(future.task_run.id)
    return running_tasks


def monitor_tasks(t):
    # print every two seconds a list of "active" and "Running" tasks for t seconds
    logger = get_run_logger()
    start = time.time()
    while time.time() - start < t:
        running_tasks = get_running_tasks()
        active_tasks = get_active_tasks()
        if running_tasks or active_tasks:
            logger.info(f"Running tasks ({len(running_tasks)}): {running_tasks}")
            logger.info(f"Active tasks ({len(active_tasks)}): {active_tasks}")
            time.sleep(2)
        else:
            break


@task
def sleep_task(t):
    # sleep for t seconds
    time.sleep(t)


@flow
def test_limits():
    logger = get_run_logger()

    # setting concurrency limit 5 for tag
    create_limit(5)

    # submit 20 parallel tagged tasks that sleep for 20s each
    with tags(tag):
        for _ in range(0, 20):
            sleep_task.submit(20)
            # time.sleep(0.2) # This leads to expected behaviour

    # monitor "Running" and "active" tasks for 30s
    # see issue 1 in output: more tasks "Running" than "active"
    monitor_tasks(30)

    # increase concurrency limit while tasks are still running
    logger.info("Updating limits")
    create_limit(10)

    # continue monitoring
    # see issue 2 in output: No tasks in "Running" but still tasks in "active"
    monitor_tasks(120)

    # After the flow exits, check limits via `prefect concurrency-limit inspect test-limit`
    # see that there are still "Active Task Runs" even though the flow "Finished in state Completed('All states completed.')"


if __name__ == "__main__":
    test_limits()

Error

17:01:35.428 | INFO    | prefect.engine - Created flow run 'cherry-orangutan' for flow 'test-limits'
17:01:37.039 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-0' for task 'sleep_task'
17:01:37.040 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-0' for execution.
17:01:37.113 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-2' for task 'sleep_task'
17:01:37.115 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-2' for execution.
17:01:37.126 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-8' for task 'sleep_task'
17:01:37.127 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-8' for execution.
17:01:37.138 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-1' for task 'sleep_task'
17:01:37.139 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-1' for execution.
17:01:37.143 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-4' for task 'sleep_task'
17:01:37.144 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-4' for execution.
17:01:37.154 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-13' for task 'sleep_task'
17:01:37.155 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-13' for execution.
17:01:37.160 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-11' for task 'sleep_task'
17:01:37.161 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-11' for execution.
17:01:37.164 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-10' for task 'sleep_task'
17:01:37.165 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-10' for execution.
17:01:37.166 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-7' for task 'sleep_task'
17:01:37.167 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-7' for execution.
17:01:37.169 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-3' for task 'sleep_task'
17:01:37.170 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-3' for execution.
17:01:37.174 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-19' for task 'sleep_task'
17:01:37.174 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-19' for execution.
17:01:37.179 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-6' for task 'sleep_task'
17:01:37.180 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-6' for execution.
17:01:37.182 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-16' for task 'sleep_task'
17:01:37.183 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-16' for execution.
17:01:37.186 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-9' for task 'sleep_task'
17:01:37.186 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-9' for execution.
17:01:37.192 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-17' for task 'sleep_task'
17:01:37.193 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-17' for execution.
17:01:37.199 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-15' for task 'sleep_task'
17:01:37.199 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-15' for execution.
17:01:37.201 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-12' for task 'sleep_task'
17:01:37.202 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-12' for execution.
17:01:37.206 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-18' for task 'sleep_task'
17:01:37.206 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-18' for execution.
17:01:37.456 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-14' for task 'sleep_task'
17:01:37.456 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-14' for execution.
17:01:37.460 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-5' for task 'sleep_task'
17:01:37.461 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-5' for execution.
17:01:41.672 | INFO    | Flow run 'cherry-orangutan' - Running tasks (17): [UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:41.672 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:48.066 | INFO    | Flow run 'cherry-orangutan' - Running tasks (18): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:48.067 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:54.367 | INFO    | Flow run 'cherry-orangutan' - Running tasks (18): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:54.368 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:57.844 | INFO    | Task run 'sleep_task-f305a62b-0' - Finished in state Completed()
17:01:57.959 | INFO    | Task run 'sleep_task-f305a62b-8' - Finished in state Completed()
17:01:57.963 | INFO    | Task run 'sleep_task-f305a62b-1' - Finished in state Completed()
17:01:57.968 | INFO    | Task run 'sleep_task-f305a62b-2' - Finished in state Completed()
17:01:57.975 | INFO    | Task run 'sleep_task-f305a62b-11' - Finished in state Completed()
17:01:58.007 | INFO    | Task run 'sleep_task-f305a62b-16' - Finished in state Completed()
17:01:58.023 | INFO    | Task run 'sleep_task-f305a62b-10' - Finished in state Completed()
17:01:58.025 | INFO    | Task run 'sleep_task-f305a62b-15' - Finished in state Completed()
17:01:58.030 | INFO    | Task run 'sleep_task-f305a62b-9' - Finished in state Completed()
17:01:58.032 | INFO    | Task run 'sleep_task-f305a62b-7' - Finished in state Completed()
17:01:58.044 | INFO    | Task run 'sleep_task-f305a62b-13' - Finished in state Completed()
17:01:58.056 | INFO    | Task run 'sleep_task-f305a62b-18' - Finished in state Completed()
17:01:58.059 | INFO    | Task run 'sleep_task-f305a62b-6' - Finished in state Completed()
17:01:58.110 | INFO    | Task run 'sleep_task-f305a62b-19' - Finished in state Completed()
17:01:58.112 | INFO    | Task run 'sleep_task-f305a62b-4' - Finished in state Completed()
17:01:58.132 | INFO    | Task run 'sleep_task-f305a62b-3' - Finished in state Completed()
17:01:58.387 | INFO    | Task run 'sleep_task-f305a62b-12' - Finished in state Completed()
17:01:58.405 | INFO    | Task run 'sleep_task-f305a62b-17' - Finished in state Completed()
17:02:00.350 | INFO    | Flow run 'cherry-orangutan' - Running tasks (7): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca')]
17:02:00.351 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:06.653 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:06.654 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:08.657 | INFO    | Flow run 'cherry-orangutan' - Updating limits
17:02:12.907 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:12.908 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:18.628 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:18.629 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:25.434 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:25.434 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:28.555 | INFO    | Task run 'sleep_task-f305a62b-14' - Finished in state Completed()
17:02:28.787 | INFO    | Task run 'sleep_task-f305a62b-5' - Finished in state Completed()
17:02:31.648 | INFO    | Flow run 'cherry-orangutan' - Running tasks (1): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:31.649 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:38.202 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:38.204 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:44.344 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:44.345 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:50.285 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:50.286 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:56.429 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:56.430 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:02.778 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:02.779 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:09.998 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:09.998 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:16.047 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:16.047 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:22.320 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:22.321 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:28.480 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:28.482 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:34.559 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:34.560 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:40.872 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:40.874 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:47.528 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:47.529 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:53.361 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:53.361 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:00.123 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:00.124 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:06.829 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:06.830 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:13.025 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:13.026 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:15.278 | INFO    | Flow run 'cherry-orangutan' - Finished in state Completed('All states completed.')

Versions

Version:             2.6.1
API version:         0.8.2
Python version:      3.8.10
Git commit:          d9dd4443
Built:               Fri, Oct 14, 2022 2:01 PM
OS/Arch:             linux/x86_64
Server type:         cloud

Additional context

relates to #5995

Oct 17 '22 14:10 j-tr

@anticorrelator is it possible this is a regression in the new concurrency limit implementation?

@j-tr have you encountered this in the open source version?

Oct 17 '22 22:10 zanieb

That update isn't live yet, but will be soon--I think this error seems consistent with a previous issue that I believe we're fixing with the upcoming release

Oct 17 '22 22:10 anticorrelator

Still seeing some runs listed as active.

Same code as above:

(base) jeffhale demos/shelling $ python test_con.py    
14:34:06.199 | INFO    | prefect.engine - Created flow run 'astute-roadrunner' for flow 'test-limits'
14:34:07.017 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-0' for task 'sleep_task'
14:34:07.017 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-0' for execution.
14:34:07.056 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-2' for task 'sleep_task'
14:34:07.056 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-2' for execution.
14:34:07.058 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-7' for task 'sleep_task'
14:34:07.058 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-7' for execution.
14:34:07.061 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-6' for task 'sleep_task'
14:34:07.061 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-6' for execution.
14:34:07.063 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-14' for task 'sleep_task'
14:34:07.063 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-14' for execution.
14:34:07.064 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-4' for task 'sleep_task'
14:34:07.064 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-4' for execution.
14:34:07.066 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-8' for task 'sleep_task'
14:34:07.066 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-8' for execution.
14:34:07.069 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-1' for task 'sleep_task'
14:34:07.069 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-1' for execution.
14:34:07.070 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-18' for task 'sleep_task'
14:34:07.071 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-18' for execution.
14:34:07.072 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-9' for task 'sleep_task'
14:34:07.072 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-9' for execution.
14:34:07.073 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-17' for task 'sleep_task'
14:34:07.074 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-17' for execution.
14:34:07.079 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-3' for task 'sleep_task'
14:34:07.079 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-3' for execution.
14:34:07.080 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-10' for task 'sleep_task'
14:34:07.081 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-10' for execution.
14:34:07.085 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-12' for task 'sleep_task'
14:34:07.085 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-12' for execution.
14:34:07.088 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-5' for task 'sleep_task'
14:34:07.088 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-5' for execution.
14:34:07.091 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-16' for task 'sleep_task'
14:34:07.091 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-16' for execution.
14:34:07.105 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-11' for task 'sleep_task'
14:34:07.105 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-11' for execution.
14:34:07.111 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-13' for task 'sleep_task'
14:34:07.111 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-13' for execution.
14:34:07.137 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-15' for task 'sleep_task'
14:34:07.138 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-15' for execution.
14:34:07.478 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-19' for task 'sleep_task'
14:34:07.478 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-19' for execution.
14:34:08.821 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:08.822 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:12.412 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:12.412 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:16.046 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:16.048 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:19.805 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:19.806 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:23.408 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:23.409 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:26.605 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:26.606 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:27.311 | INFO    | Task run 'sleep_task-f305a62b-0' - Finished in state Completed()
14:34:27.356 | INFO    | Task run 'sleep_task-f305a62b-1' - Finished in state Completed()
14:34:27.363 | INFO    | Task run 'sleep_task-f305a62b-2' - Finished in state Completed()
14:34:27.371 | INFO    | Task run 'sleep_task-f305a62b-11' - Finished in state Completed()
14:34:27.436 | INFO    | Task run 'sleep_task-f305a62b-7' - Finished in state Completed()
14:34:29.698 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:29.699 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:32.999 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:33.000 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:36.556 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:36.556 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:38.561 | INFO    | Flow run 'astute-roadrunner' - Updating limits
14:34:39.765 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:39.767 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:43.259 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:43.260 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:46.397 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:46.398 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:49.683 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:49.685 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:53.226 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:53.227 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:56.883 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:56.884 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:57.416 | INFO    | Task run 'sleep_task-f305a62b-3' - Finished in state Completed()
14:34:57.490 | INFO    | Task run 'sleep_task-f305a62b-12' - Finished in state Completed()
14:34:57.492 | INFO    | Task run 'sleep_task-f305a62b-17' - Finished in state Completed()
14:34:57.495 | INFO    | Task run 'sleep_task-f305a62b-18' - Finished in state Completed()
14:35:00.431 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:00.433 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:03.724 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:03.726 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:06.919 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:06.920 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:10.189 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:10.190 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:13.690 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:13.691 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:17.042 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:17.043 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:20.494 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:20.496 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:23.782 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:23.783 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:27.070 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:27.072 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:27.567 | INFO    | Task run 'sleep_task-f305a62b-6' - Finished in state Completed()
14:35:27.680 | INFO    | Task run 'sleep_task-f305a62b-10' - Finished in state Completed()
14:35:27.807 | INFO    | Task run 'sleep_task-f305a62b-15' - Finished in state Completed()
14:35:27.814 | INFO    | Task run 'sleep_task-f305a62b-8' - Finished in state Completed()
14:35:27.846 | INFO    | Task run 'sleep_task-f305a62b-9' - Finished in state Completed()
14:35:27.962 | INFO    | Task run 'sleep_task-f305a62b-13' - Finished in state Completed()
14:35:28.016 | INFO    | Task run 'sleep_task-f305a62b-19' - Finished in state Completed()
14:35:30.286 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:30.287 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:33.579 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:33.580 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:37.463 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:37.465 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:40.743 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:40.744 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:44.317 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:44.318 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:47.331 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:47.332 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:50.434 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:50.435 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:53.501 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:53.502 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:56.552 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:56.553 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:58.095 | INFO    | Task run 'sleep_task-f305a62b-14' - Finished in state Completed()
14:35:58.172 | INFO    | Task run 'sleep_task-f305a62b-16' - Finished in state Completed()
14:35:58.266 | INFO    | Task run 'sleep_task-f305a62b-5' - Finished in state Completed()
14:35:58.279 | INFO    | Task run 'sleep_task-f305a62b-4' - Finished in state Completed()
14:35:59.680 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:59.682 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:03.590 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:03.591 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:06.703 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:06.704 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:09.958 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:09.959 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:13.437 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:13.438 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:16.795 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:16.796 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:20.011 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:20.012 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:23.490 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:23.490 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:26.578 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:26.578 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:30.126 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:30.128 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:34.116 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:34.117 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:37.641 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:37.642 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:39.722 | INFO    | Flow run 'astute-roadrunner' - Finished in state Completed('All states completed.')

Oct 18 '22 18:10 discdiver

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

Mar 30 '23 18:03 github-actions[bot]

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.

Apr 13 '23 18:04 github-actions[bot]

prefect prefect copied to clipboard

Zombie Tasks Runs caused by increasing concurrency limit during task run

First check

Bug summary

Reproduction

Error

Versions

Additional context

prefect
prefect copied to clipboard