prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Generating task_key fails on function objects, as objects have no __code__ method

Open pgulley opened this issue 1 year ago • 3 comments

Bug summary

This is pretty niche- in my project , I assemble prefect flows dynamically based on some user provided configuration- this works via a common object which provides a call method, which individual tasks all subclass from- these are registered with flow(). A master "Pipeline" object follows the same pattern to actually call these flows in order to produce the This worked fine previously, but upon upgrading to prefect 2.20.3, I'm encountering a new error:

File "/Users/pgulley/Projects/system-metrics/venv/lib/python3.12/site-packages/prefect/tasks.py", line 145, in _generate_task_key h[:NUM_CHARS_DYNAMIC_KEY] if (h := hash_objects(fn.__code__)) else "unknown" ^^^^^^^^^^^ AttributeError: type object 'Pipeline' has no attribute '__code__'. Did you mean: '__doc__'? This is new as of this update- so I'm not sure what's changed, but it's clear that it has to do with this funny, maybe off-label approach I've taken (as objects don't have a code method like functions do)

(Related, I think, to this issue: https://github.com/PrefectHQ/prefect/issues/12340)

Version info (prefect version output)

Prefect version yeilds: 
Version:             2.14.20
API version:         0.8.4
Python version:      3.11.9
Git commit:          8ceb0962
Built:               Thu, Feb 1, 2024 6:30 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         cloud

But in my venv I have prefect 2.20.3 running

Additional context

No response

pgulley avatar Aug 23 '24 00:08 pgulley

I've been able to circumvent by adding a code constant to the pipeline object, but this feels like a bodge

pgulley avatar Aug 23 '24 00:08 pgulley

Thanks for the issue @pgulley and for the research you did for it! You're correct that it's definitely related to that issue - I think it was this PR specifically that triggered the problem for you.

I'm glad you have a workaround for now, but we'll also definitely get a fix for this. In fact I have an idea: can you confirm that the following attribute access works for you:

pipeline_instance = Pipeline(**kwargs)
pipeline_instance.__call__.__code__

I believe class call methods have the code object that we use for functions, so a more nuanced check of whether the fn is a true function or a callable should allow us to access it without error.

cicdw avatar Aug 23 '24 03:08 cicdw

Great! Thanks for the prompt response! Yes- that code you gave works- the call method does have the code object you need!

pgulley avatar Aug 26 '24 18:08 pgulley