flyteorg
Pickle as a default Transformer rather than a special case of function signatures
TL;DR
Right now only "top level" missing types (those defined directly in a function) are pickled if they don't have a registered TypeTransformer. This PR alters that behavior to anywhere TypeTransformers are used.
Type
- [ ] Bug Fix
- [X ] Feature
- [ ] Plugin
Are all requirements met?
- [ ] Code completed
- [ ] Smoke tested
- [ ] Unit tests added
- [ ] Code documentation added
- [ ] Any pending items have an associated Issue
Complete description
The special casing for pickling as a serialization/deserialization method leads to some strange behavior where, e.g
@task
def my_task(foo: Foo):
...
Is valid, because that type is converted, but e.g. Foo inside of a dataclass etc is not. Because the conversion to the placeholder FlytePickle type happens in a function which is only called once on top level type, and only traverses the origin stack, rather than being used anywhere TypeTransformers are used. This PR fixes that problem, while also significantly simplifying the logic to support it.
Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue
NA
OR
https://github.com/flyteorg/flyte/issues/
