adding attribute and pickling causes failure when subtracting delta

[dlarosa11]

trafficstars

When I pickle the diff and then unpickle and try to subtract delta created from diff, I get error. I added prinst statements to your code starting at 'Line 571 of delta.py'.

Code:

from deepdiff import DeepDiff, Delta

def execute_with_pickle(t1, t2):
    print(f"t1: {t1}")
    print(f"t2: {t2}")
    diff = DeepDiff(t1, t2)
    jsoned = diff.to_json_pickle()
    diff_new = DeepDiff.from_json_pickle(jsoned)
    delta: Delta = Delta(diff_new, bidirectional=True)
    t3 = t2 - delta
    print(f"t3: {t3}")

def execute_without_pickle(t1, t2):
    print(f"t1: {t1}")
    print(f"t2: {t2}")
    diff = DeepDiff(t1, t2)
    delta: Delta = Delta(diff, bidirectional=True)
    t3 = t2 - delta
    print(f"t3: {t3}")

print("GOOD with pickle")
t1 = {1: 1, 2: 2, 3: 3}
t2 = {1: 1, 2: "2", 3: 3}
execute_with_pickle(t1, t2)

print("---------")

print("GOOD without pickle")
t1 = {1: 1, 2: 2, 3: 3}
t2 = {1: 1, 2: "2", 3: 3}
execute_without_pickle(t1, t2)

print("---------")

print("GOOD without pickle")
t1 = {1: 1, 2: 2, 3: 3}
t2 = {1: 1, 2: 2, 3: 3, 4: 4}
execute_without_pickle(t1, t2)

print("---------")

print("ERROR with pickle")
t1 = {1: 1, 2: 2, 3: 3}
t2 = {1: 1, 2: 2, 3: 3, 4: 4}
execute_with_pickle(t1, t2)

Output

GOOD with pickle
t1: {1: 1, 2: 2, 3: 3}
t2: {1: 1, 2: '2', 3: 3}
t3: {1: 1, 2: 2, 3: 3}
---------
GOOD without pickle
t1: {1: 1, 2: 2, 3: 3}
t2: {1: 1, 2: '2', 3: 3}
t3: {1: 1, 2: 2, 3: 3}
---------
GOOD without pickle
t1: {1: 1, 2: 2, 3: 3}
t2: {1: 1, 2: 2, 3: 3, 4: 4}
Line 571 of delta.py
type(items): <class 'dict'>
items: {'root[4]': 4}
t3: {1: 1, 2: 2, 3: 3}
---------
ERROR with pickle
t1: {1: 1, 2: 2, 3: 3}
t2: {1: 1, 2: 2, 3: 3, 4: 4}
Line 571 of delta.py
type(items): <class 'deepdiff.model.PrettyOrderedSet'>
items: [root[4]]
Traceback (most recent call last):
  File "/Users/domeniclarosa/ws/virtual_envs/myenv.deepdiff/lib/python3.9/site-packages/deepdiff/delta.py", line 574, in _do_item_removed
    sorted_item = sorted(items.items(), key=self._sort_key_for_item_added, reverse=True)
TypeError: 'list' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/domeniclarosa/ws/work/dlc/experimental/deepdifferror.py", line 45, in <module>
    execute_with_pickle(t1, t2)
  File "/Users/domeniclarosa/ws/work/dlc/experimental/deepdifferror.py", line 10, in execute_with_pickle
    t3 = t2 - delta
  File "/Users/domeniclarosa/ws/virtual_envs/myenv.deepdiff/lib/python3.9/site-packages/deepdiff/delta.py", line 209, in __rsub__
    result = self.__add__(other)
  File "/Users/domeniclarosa/ws/virtual_envs/myenv.deepdiff/lib/python3.9/site-packages/deepdiff/delta.py", line 192, in __add__
    self._do_dictionary_item_removed()
  File "/Users/domeniclarosa/ws/virtual_envs/myenv.deepdiff/lib/python3.9/site-packages/deepdiff/delta.py", line 681, in _do_dictionary_item_removed
    self._do_item_removed(dictionary_item_removed)
  File "/Users/domeniclarosa/ws/virtual_envs/myenv.deepdiff/lib/python3.9/site-packages/deepdiff/delta.py", line 576, in _do_item_removed
    sorted_item = sorted(items.items(), key=cmp_to_key(self._sort_comparison), reverse=True)
TypeError: 'list' object is not callable

[dlarosa11]

Thanks for taking a look! Dom

[seperman]

Hi @dlarosa11 Thanks for reporting the bug. I will take a look when I have a chance! PRs are always very welcome too. Also please include which version of DeepDiff you are using.

[dlarosa11]

using deepdiff==7.0.1

[dlarosa11]

using deepdiff==8.0.1 now

It seems the issue is that DeepDiff.from_json_pickle returns a dict and not a DeepDiff. I don't know how to turn the dict back into a DeepDiff with all the information it had before pickling.

Test code:

from deepdiff import DeepDiff, Delta

t1 = {1: "one"}
t2 = {1: "one", 2: "two"}

diff_before = DeepDiff(t1, t2)
print(f"diff type before pickle: {type(diff_before)}")
jsoned = diff_before.to_json_pickle()
diff = DeepDiff.from_json_pickle(jsoned)
print(f"diff type after depickle: {type(diff)} {diff}")

Output:

diff type before pickle: <class 'deepdiff.diff.DeepDiff'>
diff type after depickle: <class 'dict'> {'dictionary_item_added': ['root[2]']}

[seperman]

@dlarosa11 If you are storing deltas, you should serialize delta, not DeepDiff's result. https://zepworks.com/deepdiff/current/delta.html#delta-deserializer