prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Improve future message when using the map operator

Open znicholasbrown opened this issue 5 months ago • 0 comments

Describe the current behavior

prefect==3.0.0rc19

Using this script:

from prefect import flow, task, get_run_logger
from time import sleep


@task
def Node(sleep_seconds: float = 0.1):
    sleep(sleep_seconds)


@flow
def process_node_map(sleep_seconds_per_node: float = 0.1, number_nodes: int = 10):
    Node.map([sleep_seconds_per_node for _ in range(number_nodes)])

if __name__ == '__main__':
    process_node_map()

Results in (among other things) a flood of log messages like this:

A future was garbage collected before it resolved. Please call `.wait()` or `.result()` on futures to ensure they resolve.

The message is ambiguous as to the behavior (are we dropping states/results from these task runs? this doesn't seem to be the case, but the "to ensure they resolve" message doesn't inspire confidence) and also to the resolution.

The docs spell out this behavior (I think) pretty nicely but that log points authors neither to the docs nor the code.

Describe the proposed behavior

I don't think we should log this warning at all, but there might be valid reasons to do so. If we continue to log this warning, it should link to the documentation around the .map operator and should point to the code in some way.

Example Use

Node.map([sleep_seconds_per_node for _ in range(number_nodes)])

Results in:

A future for task "Node" was garbage collected before it resolved. To use the results of task "Node", call `.wait()` or `.result()` on the return future to ensure it resolves. See: https://docs-3.prefect.io/3.0rc/develop/task-runners#mapping-over-iterables for details.

Additional context

None

znicholasbrown avatar Aug 26 '24 15:08 znicholasbrown