Review pickling package options: e.g. dill as a replacement for cloudpickle

[satra]

https://pypi.org/project/dill/ may allow us to pickle functions that behave the same way across two different environments (by getting source code). this should allow us to pickle arbitrary functions but will have to see if multiple functions can be encoded. there is an open issue using it inside ipython (https://github.com/uqfoundation/dill/issues/346)

[nicolocin]

Was looking into typing and found out that dill doesn't preserve type annotations when pickling typed functions (uqfoundation/dill#179). Still not working on python 3.7.7 and dill 0.3.2, which is the latest version. Potentially an issue?

import dill
import typing as ty

def doit(x: int, y: ty.List[int]) -> ty.List[int]:
    return y + [x]

_doit = dill.copy(doit)

# >>> {'x': <class 'int'>, 'y': ty.List[int], 'return': ty.List[int]}
print(doit.__annotations__)

# >>> {}
print(_doit.__annotations__)

[satra]

@nicolocin - that may not be a necessary element since our use of dill is after the function has been transformed to a task, at which point it would be nice to have the annotations, but not necessary. you can check if dill can get the source code with the annotations.