Pickle or similar a marshmallow schema

[lynochka]

Hei, how could I pickle a class similar to

class SchemaMarshmallow(Schema): flag = fields.Bool() choice = fields.Str()

or

class MySchema(NamedTuple): flag: bool choice: str class SchemaMarshmallow(AnnotationSchema): class Meta: target = MySchema register_as_scheme = True

Thank you! My original problem is that i try to use marshmallow in pyspark to validate a schema of a column with MapType and it throws a PicklingError: Can't pickle typing.Union[bool, NoneType]: it's not the same object as typing.Union

[justanr]

I'm honestly not sure the best way to proceed here. I've largely stayed away from pickling myself, aside from a handful of things at my last employer, so I'm not sure what would cause the disconnect.

This could be an issue with the version of Python that the spark cluster is using (Union and Optional have be implemented different ways in different versions of Python), from just pickling and unpickling locally it doesn't seem to cause an issue, but I'm also just doing a round-robin in a jupyter notebook (done in python3.6.5 since that's what the existing venv for this project I had setup used):

import pickle
from typing import NamedTuple, Optional
from marshmallow_annotations import AnnotationSchema

class MySchema(NamedTuple):
    flag: Optional[bool]
    choice: str
        
class SchemaMarshmallow(AnnotationSchema):
    class Meta:
        target = MySchema
        register_as_scheme = True
        
scheme = pickle.loads(pickle.dumps(SchemaMarshmallow()))
scheme.dump(MySchema(flag=None, choice="baz"))

# MarshalResult(data={'choice': 'baz', 'flag': None}, errors={})