PsyNeuLink icon indicating copy to clipboard operation
PsyNeuLink copied to clipboard

Published 20 hours ago • verified publisher icon PrincetonUniversity

error in autodiff deepcopy after 30f60b2beca71

Open jvesely opened this issue 4 years ago • 12 comments

I started seeing this after 30f60b2beca71 with Yotam's model. I don't have a smaller reproducer atm

Traceback (most recent call last):
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 786, in copy_iterable_with_shared
    raise TypeError
TypeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 786, in copy_iterable_with_shared
    raise TypeError
TypeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 359, in copy_parameter_value
    memo=memo
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 744, in copy_iterable_with_shared
    new_v = copy_iterable_with_shared(v, shared_types)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 768, in copy_iterable_with_shared
    result = obj.__class__()
TypeError: __init__() missing 1 required positional argument: 'component_type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 786, in copy_iterable_with_shared
    raise TypeError
TypeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 786, in copy_iterable_with_shared
    raise TypeError
TypeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Yotam LCA Model.py", line 689, in <module>
    perf_dicts.append(evaluate_net_perf_lca(mnet_lca, [task_i, task_j], all_tasks, num_features, num_input_dims, num_output_dims, num_test_points))
  File "Yotam LCA Model.py", line 503, in evaluate_net_perf_lca
    res = mnet_lca.run( { mnet_lca.nodes['mnet'] : input_set['inputs'] }, bin_execute=LCA_BIN_EXECUTE )
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/context.py", line 719, in wrapper
    return func(*args, context=context, **kwargs)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/compositions/composition.py", line 8292, in run
    self._initialize_from_context(context, base_context, override=False)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/components/component.py", line 2269, in _initialize_from_context
    comp._initialize_from_context(context, base_context, override, visited=visited)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/components/component.py", line 2274, in _initialize_from_context
    param._initialize_from_context(context, base_context, override)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 1470, in _initialize_from_context
    new_val = copy.deepcopy(new_val)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 776, in copy_iterable_with_shared
    new_item = copy.deepcopy(item, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/components/component.py", line 1259, in __deepcopy__
    newone = fun(self, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 719, in __deepcopy__
    res_val = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 403, in __deepcopy__
    newone = get_deepcopy_with_shared(self._deepcopy_shared_keys)(self, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 719, in __deepcopy__
    res_val = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 901, in __deepcopy__
    for k in self._param_attrs
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 901, in <dictcomp>
    for k in self._param_attrs
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 366, in copy_parameter_value
    return copy.deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/components/component.py", line 1259, in __deepcopy__
    newone = fun(self, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 748, in copy_iterable_with_shared
    new_v = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/usr/lib/python3.6/copy.py", line 273, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/components/component.py", line 1259, in __deepcopy__
    newone = fun(self, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 719, in __deepcopy__
    res_val = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 403, in __deepcopy__
    newone = get_deepcopy_with_shared(self._deepcopy_shared_keys)(self, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 719, in __deepcopy__
    res_val = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 901, in __deepcopy__
    for k in self._param_attrs
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 901, in <dictcomp>
    for k in self._param_attrs
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/parameters.py", line 359, in copy_parameter_value
    memo=memo
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 748, in copy_iterable_with_shared
    new_v = copy.deepcopy(v, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 717, in __deepcopy__
    res_val = copy_iterable_with_shared(v, shared_types, memo)
  File "/ilab/users/jv356/PsyNeuLink/psyneulink/core/globals/utilities.py", line 776, in copy_iterable_with_shared
    new_item = copy.deepcopy(item, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib/python3.6/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python3.6/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python3.6/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python3.6/copy.py", line 161, in deepcopy
    y = copier(memo)
  File "/ilab/users/jv356/.local/lib/python3.6/site-packages/torch/tensor.py", line 38, in __deepcopy__
    raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

jvesely avatar Oct 27 '20 18:10 jvesely

cc @SamKG , @kmantel

jvesely avatar Oct 27 '20 18:10 jvesely

seems like a limitation with Pytorch; is there a specific usecase for deepcopying the pytorch model? ( @kmantel )

SamKG avatar Oct 27 '20 18:10 SamKG

I added that after noticing that copying AutodiffComposition.pytorch_representation to new contexts would end up deep copying entire Compositions and other Components. Is it safe to make pytorch_representation non-stateful? On Oct 27, 2020, at 14:37, Jan Vesely <[email protected]mailto:[email protected]> wrote:

cc @SamKGhttps://github.com/SamKG , @kmantelhttps://github.com/kmantel

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/PrincetonUniversity/PsyNeuLink/issues/1808#issuecomment-717451755, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMEWO7YA246NSYCM7ZTMVLSM4HPVANCNFSM4TBHJ5UQ.

kmantel avatar Oct 27 '20 18:10 kmantel

non-stateful pytorch representations may break things, since the initial parameters of the network are copied from the current context (but are never copied again afterwards).

it should be fine to leave the parameter as None when creating a copy - AutodiffCompositions should create a new pytorch model from the existing context when it is needed.

SamKG avatar Oct 27 '20 18:10 SamKG

@jvesely, I wasn't able to replicate the error using Yotam's model from #1770, would you be able to test if these changes fix it? https://github.com/PrincetonUniversity/PsyNeuLink/compare/devel...kmantel:ec7c7490916d65fe5e2bc21e6c7e6787c9471e21 (patch)

It should fix a bug in copy_iterable_with_shared for ContentAddressableList, which is one of the exceptions above, and prevent copying of pytorch_representation.

kmantel avatar Oct 28 '20 22:10 kmantel

looks like the bug existed before 30f60b2. I can still see it after reverting said commit. It looks to be tied to the original training in get_trained_network_multLCA not using compiled form (bin_execute='Python'). It happens ~ 60% of the time. I'll try to test the patch this weekend.

jvesely avatar Oct 29 '20 00:10 jvesely

sorry for the delay. The issue is still reproducible on devel (8bf0f7ff932bd5b1cb0caef52bbd6175ba357b04) Here's a reproducer: MNET=Python python Scripts/Debug/Yotam\ LCA\ Model\ LLVM.py

The patch from comment https://github.com/PrincetonUniversity/PsyNeuLink/issues/1808#issuecomment-718244998 no longer applies.

jvesely avatar Feb 15 '21 05:02 jvesely

the bug still reproduces. after https://github.com/PrincetonUniversity/PsyNeuLink/pull/2685 the invocation line is:

MNET=PyTorch python Scripts/Debug/Yotam\ LCA\ Model\ LLVM.py

jvesely avatar Jun 02 '23 00:06 jvesely

Interestingly it doesn't seem to be failing on every run for me. Here it's failing when trying to copy a PytorchMechanismWrapper.value. I ran into this same thing recently in another set of changes, and that commit included fixing this with tensor.detach(), which I pushed on its own here: https://github.com/kmantel/PsyNeuLink/commit/pytorch-comps-copy

kmantel avatar Jun 02 '23 02:06 kmantel

the approach makes sense. Maybe we could do detach().numpy() to not consider tensor data type. I'm a bit wary of using a try block though. Hitting except is quite expensive, are all/most value members tensors?

jvesely avatar Jun 02 '23 04:06 jvesely

It looks like in our tests it only has None or torch.Tensor

kmantel avatar Jun 02 '23 21:06 kmantel

Looking further, the None might only be from the initial value set in PytorchMechanismWrapper.__init__

kmantel avatar Jun 02 '23 23:06 kmantel