AttributeError: Can't pickle local object 'DeepDiff.__init__.<locals>.<lambda>'

[behrm]

With version 5.5.0 it is perfectly possible to run deepdiff in a multiprocessing context (with ProcessPoolExecutor ....) without any problem. So thank you very much for providing this excellent tool.

Today I built a new python environment with deepdiff version 5.7.0 and obtain the following error:

File "O:\_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists
   for element in iterable:
 File "O:\_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures\_base.py", line 598, in result_iterator
   yield fs.pop().result()
 File "O:\_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures\_base.py", line 435, in result
   return self.__get_result()
 File "O:\_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures\_base.py", line 384, in __get_result
   raise self._exception
AttributeError: Can't pickle local object 'DeepDiff.__init__.<locals>.<lambda>'

types_in_groups = [
   (type(None), str),
   (type(None), float),
   (type(None), int),
   (type(None), datetime.date)]

deepdiff = DeepDiff(
                   reference.dictObject,
                   myObject.dictObject,
                   ignore_order = True,
                   ignore_type_in_groups = types_in_groups,
                   truncate_datetime = 'day')

How can I prevent this problem when calling deepdiff?

[seperman]

Hi Bernhard,

Can you please post a reproducible code example including the multiprocessing part? Thanks!

Sep Dehpour

On Feb 28, 2022, at 7:54 AM, Bernhard Ehrminger @.***> wrote:

 With version 5.5.0 it is perfectly possible to run deepdiff in a multiprocessing context (with ProcessPoolExecutor ....) without any problem. So thank you very much for providing this excellent tool.

Today I built a new python environment with deepdiff version 5.7.0 and obtain the following error:

File "O:_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures\process.py", line 483, in _chain_from_iterable_of_lists for element in iterable: File "O:_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures_base.py", line 598, in result_iterator yield fs.pop().result() File "O:_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures_base.py", line 435, in result return self.__get_result() File "O:_Ressourcen\ESRI\ArcGISPro\conda\env\arcgispro-py3-2_9\lib\concurrent\futures_base.py", line 384, in __get_result raise self._exception AttributeError: Can't pickle local object 'DeepDiff.init..'

types_in_groups = [ (type(None), str), (type(None), float), (type(None), int), (type(None), datetime.date)]

deepdiff = DeepDiff( reference.dictObject, myObject.dictObject, ignore_order = True, ignore_type_in_groups = types_in_groups, truncate_datetime = 'day') How can I prevent this problem when calling deepdiff?

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

[behrm]

hi Sep, here is some Code:

from concurrent.futures.process import ProcessPoolExecutor

from deepdiff.diff import DeepDiff


def main():
    
    t1 = [[1, 2, 3, 9], [9, 8, 5, 9]]
    t2 = [[1, 2, 4, 10], [4, 2, 5]]

    diff = DeepDiff(t1, t2)
    print(diff)
    
    futures = []
    with ProcessPoolExecutor(max_workers = 1) as executor:

        futures.append(executor.submit(DeepDiff, t1, t2))
        
        for future in as_completed(futures):
            if future._exception is None and future._result is not None:
                # DeepDiff 5.5.0
                print(future._result)
            else:
                # DeepDiff 6.3.1
                print(future._exception)
    pass


if __name__ == '__main__':
    main()

With DeepDiff 5.5.0 its output is:

{'values_changed': {'root[0][2]': {'new_value': 4, 'old_value': 3}, 'root[0][3]': {'new_value': 10, 'old_value': 9}, 'root[1][0]': {'new_value': 4, 'old_value': 9}, 'root[1][1]': {'new_value': 2, 'old_value': 8}}, 'iterable_item_removed': {'root[1][3]': 9}}
{'values_changed': {'root[0][2]': {'new_value': 4, 'old_value': 3}, 'root[0][3]': {'new_value': 10, 'old_value': 9}, 'root[1][0]': {'new_value': 4, 'old_value': 9}, 'root[1][1]': {'new_value': 2, 'old_value': 8}}, 'iterable_item_removed': {'root[1][3]': 9}}

With DeepDiff 6.3.1 its output is:

{'values_changed': {'root[0][2]': {'new_value': 4, 'old_value': 3}, 'root[0][3]': {'new_value': 10, 'old_value': 9}, 'root[1][0]': {'new_value': 4, 'old_value': 9}, 'root[1][1]': {'new_value': 2, 'old_value': 8}}, 'iterable_item_removed': {'root[1][3]': 9}}
Can't pickle local object 'DeepDiff.__init__.<locals>.<lambda>'

[seperman]

Hi @behrm I fixed this in the dev branch. I will keep you posted once I cut a new release.

[seperman]

This was released back in September.