prefect
prefect copied to clipboard
Improve future message when using the map operator
Describe the current behavior
prefect==3.0.0rc19
Using this script:
from prefect import flow, task, get_run_logger
from time import sleep
@task
def Node(sleep_seconds: float = 0.1):
sleep(sleep_seconds)
@flow
def process_node_map(sleep_seconds_per_node: float = 0.1, number_nodes: int = 10):
Node.map([sleep_seconds_per_node for _ in range(number_nodes)])
if __name__ == '__main__':
process_node_map()
Results in (among other things) a flood of log messages like this:
A future was garbage collected before it resolved. Please call `.wait()` or `.result()` on futures to ensure they resolve.
The message is ambiguous as to the behavior (are we dropping states/results from these task runs? this doesn't seem to be the case, but the "to ensure they resolve" message doesn't inspire confidence) and also to the resolution.
The docs spell out this behavior (I think) pretty nicely but that log points authors neither to the docs nor the code.
Describe the proposed behavior
I don't think we should log this warning at all, but there might be valid reasons to do so. If we continue to log this warning, it should link to the documentation around the .map
operator and should point to the code in some way.
Example Use
Node.map([sleep_seconds_per_node for _ in range(number_nodes)])
Results in:
A future for task "Node" was garbage collected before it resolved. To use the results of task "Node", call `.wait()` or `.result()` on the return future to ensure it resolves. See: https://docs-3.prefect.io/3.0rc/develop/task-runners#mapping-over-iterables for details.
Additional context
None