aiida-core
aiida-core copied to clipboard
Published
20 hours ago ā¢
aiidateam
aiidateam
"maximum recursion depth reached" when submitting ~200 workchains
@danieleongari reports the following error when submitting ~200 workchains with aiida-core 1.6.1
2021-04-27 16:41:13 [26387 | ERROR]: Traceback (most recent call last):
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/manage/external/rmq.py", line 208, in _continue
result = await super()._continue(communicator, pid, nowait, tag)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/process_comms.py", line 607, in _continue
proc = cast('Process', saved_state.unbundle(self._load_context))
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/persistence.py", line 60, in unbundle
return Savable.load(self, load_context)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/persistence.py", line 452, in load
return load_cls.recreate_from(saved_state, load_context)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/processes.py", line 238, in recreate_from
process = cast(Process, super().recreate_from(saved_state, load_context))
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/persistence.py", line 477, in recreate_from
call_with_super_check(obj.load_instance_state, saved_state, load_context)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/engine/processes/workchains/workchain.py", line 105, in load_instance_state
super().load_instance_state(saved_state, load_context)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/engine/processes/process.py", line 284, in load_instance_state
super().load_instance_state(saved_state, load_context)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/plumpy/processes.py", line 620, in load_instance_state
decoded = self.decode_input_args(saved_state[BundleKeys.INPUTS_RAW])
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/engine/processes/process.py", line 607, in decode_input_args
return serialize.deserialize(encoded)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/serialize.py", line 230, in deserialize
return yaml.load(serialized, Loader=AiiDALoader)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
return loader.get_single_data()
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 43, in get_single_data
return self.construct_document(node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 47, in construct_document
data = self.construct_object(node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 92, in construct_object
data = constructor(self, node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/serialize.py", line 131, in mapping_constructor
yaml_node = loader.construct_mapping(mapping, deep=True)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 210, in construct_mapping
return super().construct_mapping(node, deep=deep)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 135, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 99, in construct_object
for dummy in generator:
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 404, in construct_yaml_map
value = self.construct_mapping(node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 210, in construct_mapping
return super().construct_mapping(node, deep=deep)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 135, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/yaml/constructor.py", line 92, in construct_object
data = constructor(self, node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/serialize.py", line 56, in node_constructor
return orm.load_node(uuid=yaml_node)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/__init__.py", line 197, in load_node
return load_entity(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/__init__.py", line 77, in load_entity
return entity_loader.load_entity(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/utils/loaders.py", line 213, in load_entity
entity = builder.one()[0]
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 2179, in one
res = self.all()
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 2252, in all
matches = list(self.iterall(batch_size=batch_size))
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 2209, in iterall
query = self.get_query()
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 2088, in get_query
query = self._build()
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 1938, in _build
self._query = self._query.filter(self._build_filters(alias, filter_specs))
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/querybuilder.py", line 1373, in _build_filters
self._impl.get_filter_expr(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/implementation/django/querybuilder.py", line 212, in get_filter_expr
self.get_filter_expr(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/implementation/django/querybuilder.py", line 237, in get_filter_expr
expr = self.get_filter_expr_from_column(operator, value, column)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/aiida/orm/implementation/querybuilder.py", line 217, in get_filter_expr_from_column
expr = database_entity.cast(String).like(value)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 236, in __getattr__
return getattr(self.comparator, key)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 987, in __getattr__
return self._fallback_getattr(key)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/orm/properties.py", line 364, in _fallback_getattr
return getattr(self.__clause_element__(), key)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 974, in oneshot
result = fn(*args, **kw)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/orm/properties.py", line 316, in _memoized_method___clause_element__
return self.adapter(self.prop.columns[0])
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/orm/util.py", line 680, in _adapt_element
return self._adapter.traverse(elem)._annotate(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/util.py", line 936, in traverse
return self.columns[obj]
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/util/_collections.py", line 745, in __missing__
self[key] = val = self.creator(self.weakself(), key)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/util.py", line 943, in _locate_col
c = ClauseAdapter.traverse(self, col)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 240, in traverse
return replacement_traverse(obj, self.__traverse_options__, replace)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 484, in replacement_traverse
obj = clone(obj, **opts)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 473, in clone
newelem = replace(elem)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/visitors.py", line 236, in replace
e = v.replace(elem)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/util.py", line 848, in replace
return self._corresponding_column(col, True)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/util.py", line 820, in _corresponding_column
newcol = self.selectable.corresponding_column(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 560, in corresponding_column
if self.c.contains_column(column):
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 893, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 647, in columns
self._populate_column_collection()
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 1393, in _populate_column_collection
col._make_proxy(self)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 1802, in _make_proxy
c = self._constructor(
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 1568, in __init__
self._init_items(*args)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 121, in _init_items
spwd(self)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/base.py", line 461, in _set_parent_with_dispatch
self._set_parent(parent)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 2282, in _set_parent
self.parent._on_table_attach(self._set_table)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 1722, in _on_table_attach
event.listen(self, "after_parent_attach", fn)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/api.py", line 102, in listen
_event_key(target, identifier, fn).listen(*args, **kw)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/api.py", line 25, in _event_key
tgt = evt_cls._accept_with(target)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/base.py", line 232, in _accept_with
if hasattr(target, "dispatch"):
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/base.py", line 298, in __get__
obj.__dict__["dispatch"] = disp = self.dispatch._for_instance(obj)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/base.py", line 121, in _for_instance
return self._for_class(instance_cls)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/base.py", line 117, in _for_class
return self.__class__(self, instance_cls)
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/site-packages/sqlalchemy/event/base.py", line 83, in __init__
self._empty_listeners = self._empty_listener_reg[instance_cls]
File "/home/daniele/anaconda3/envs/aiida_py38/lib/python3.8/weakref.py", line 383, in __getitem__
return self.data[ref(key)]
RecursionError: maximum recursion depth exceeded while calling a Python object
the error goes away when adding a time.sleep(2) in between submissions
Your environment
- Operating system [e.g. Linux]: ubuntu
- Python version [e.g. 3.7.1]: 3.8
- aiida-core version [e.g. 1.2.1]: 1.6.1
Does this come from the submission script or the daemon? Looks to be on the daemon side, correct?
Sorry, forgot to mention: it comes from the submission script, not from the daemon logs
It may also be helpful to provide the output of pip freeze, I.e to check the version of sqlalchemy
Sorry, forgot to mention: it comes from the submission script
Are you sure? Is he running the workchains in that script or sending them to the daemon? Reason that I am asking is that the stack trace shows that the problem originates from the ProcessLauncher._continue_ call which should not be called during submission. This is the hook that is called on a daemon runner when it receives a task to continue a process from RabbitMQ.
@sphuber The error comes from inspecting the report (verdi process report). The workchain is submitted from a Jupyter notebook using the builder, and then submitting it. Of the ca. 200 calculations, the first 10ish run ok, and then the most of them start to be excepted with the error that Leo reported: it looks like there is a flooding of processes that causes problems.
@chrisjsewell is this enough?
~ pip freeze | egrep sql
SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1612225077951/work
SQLAlchemy-Utils @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy-utils_1614043858099/work
sqlparse @ file:///home/conda/feedstock_root/build_artifacts/sqlparse_1602142927465/work
Thank you for the help!
Thanks for the additional info @danieleongari . It would be very useful if we actually could get the version of sqlalchemy. Maybe you can do the following in a shell or notebook:
In [1]: import sqlalchemy
In [2]: sqlalchemy.__version__
Out[2]: '1.3.23'
So as expected, the problem is on the daemon worker side and not the submit script. What is happening is that the daemon worker receives a task from the process queue from RabbitMQ, it loads the corresponding node from the database and then uses the YAML dump in the checkpoint attribute, to reconstruct the Process instance in memory from that serialized version. This includes the entire set of inputs of the process that were also serialized and so each of those are reloaded from the database, which happens in the line:
return orm.load_node(uuid=yaml_node)
from the aiida.orm.utils.serialize module. The load_node call will use the QueryBuilder underneath which will then go into sqlalchemy which has a big part in the stack trace. This is where I lose the thread, because apparently somewhere in the sqlalchemy code that is invoked when we call load_node leads to this infinite recursion. I have no idea why or why this should be related to many processes being run.
The only hint there is is that the final method in the stacktrace comes from the weakref built in module. At this point I have to start speculating but it could be that the daemon worker is trying to load a node that is already loaded in its memory (as part of the inputs of another process that it is working on, they could share the same Code as input for example) and there might be some caching mechanism in place that when the same node gets loaded, it is coupled to the existing ref instead. I am really spitballing here and have no idea how to further debug this or try and reproduce.
In [1]: import sqlalchemy
In [2]: sqlalchemy.__version__
Out[2]: '1.3.23'
sorry for the hickkup with the submission script vs daemon - there was a miscommunication on our side
I'm having a somewhat similar issue, so I'll add my case to this one. When submitting a 100 PwBaseWorkChains in a loop (without any pause in between š¬ ), half of them got stuck in the "Running" state:
$ verdi process list -S running
PK Created Process label Process State Process status
----- --------- --------------- --------------- ----------------
36774 1h ago PwBaseWorkChain āµ Running
36783 1h ago PwBaseWorkChain āµ Running
...
37166 1h ago PwBaseWorkChain āµ Running
Total results: 49
(Note that I've already deleted and restarted one manually). The logs for these are completely empty:
$ verdi process report 37156
No log messages recorded for this entry
But a bit of digging through the daemon logs gives me the following trace:
05/03/2021 08:52:17 PM <16823> kiwipy.rmq.tasks: [ERROR] Exception occurred while processing task.
Traceback (most recent call last):
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/utils.py", line 128, in __getattr__
return self[attr]
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/utils.py", line 85, in __getitem__
return self._dict[key]
KeyError: 'kpoints'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mbercx/envs/aiida-sirius/code/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 247, in validate_kpoints
kpoints = self.inputs.kpoints
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/utils.py", line 131, in __getattr__
raise AttributeError(errmsg)
AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints'
These two are basically repeated until a RecursionError is raised:
Traceback (most recent call last):
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/kiwipy/rmq/tasks.py", line 166, in _on_task
result = await result
File "/usr/lib/python3.8/asyncio/futures.py", line 257, in __await__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
future.result()
File "/usr/lib/python3.8/asyncio/futures.py", line 175, in result
raise self._exception
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/kiwipy/rmq/threadcomms.py", line 253, in done
result = kiwi_future.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/kiwipy/futures.py", line 54, in capture_exceptions
yield
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/communications.py", line 48, in on_done
result = plum_future.result()
File "/usr/lib/python3.8/asyncio/futures.py", line 175, in result
raise self._exception
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/kiwipy/futures.py", line 54, in capture_exceptions
yield
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/futures.py", line 73, in run_task
res = await coro()
File "/home/mbercx/.virtualenvs/aiida-sirius/lib/python3.8/site-packages/plumpy/process_comms.py", line 539, in __call__
return await self._continue(communicator, **task.get(TASK_ARGS, {}))
File "/home/mbercx/envs/aiida-sirius/code/aiida-core/aiida/manage/external/rmq.py", line 219, in _continue
self.handle_continue_exception(node, exception, message)
File "/home/mbercx/envs/aiida-sirius/code/aiida-core/aiida/manage/external/rmq.py", line 158, in handle_continue_exception
node.logger.exception(message)
File "/usr/lib/python3.8/logging/__init__.py", line 1814, in exception
self.log(ERROR, msg, *args, exc_info=exc_info, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1829, in log
self.logger.log(level, msg, *args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1500, in log
self._log(level, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1577, in _log
self.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1587, in handle
self.callHandlers(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
hdlr.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 950, in handle
self.emit(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1081, in emit
msg = self.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 925, in format
return fmt.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 672, in format
record.exc_text = self.formatException(record.exc_info)
File "/usr/lib/python3.8/logging/__init__.py", line 622, in formatException
traceback.print_exception(ei[0], ei[1], tb, None, sio)
File "/usr/lib/python3.8/traceback.py", line 103, in print_exception
for line in TracebackException(
File "/usr/lib/python3.8/traceback.py", line 493, in __init__
context = TracebackException(
File "/usr/lib/python3.8/traceback.py", line 493, in __init__
context = TracebackException(
File "/usr/lib/python3.8/traceback.py", line 493, in __init__
context = TracebackException(
[Previous line repeated 34 more times]
File "/usr/lib/python3.8/traceback.py", line 476, in __init__
_seen.add(id(exc_value))
RecursionError: maximum recursion depth exceeded while calling a Python object
After deleting one and restarting it, it ran just fine. I've also started ~35 work chains like this just fine. I then tried deleting the remaining 49, and restarting them all at once, and then I wind up with 7 PwBaseWorkChains that are stuck in Running, with the same error trace as above, after which is a whole range of reports of launching the PwCalculations:
RecursionError: maximum recursion depth exceeded while calling a Python object
05/03/2021 10:28:13 PM <17394> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [37947|PwBaseWorkChain|run_process]: launching PwCalculation<37961> iteration #1
05/03/2021 10:28:14 PM <17394> aiida.orm.nodes.process.workflow.workchain.WorkChainNode: [REPORT] [37849|PwBaseWorkChain|run_process]: launching PwCalculation<37964> iteration #1
...
Afterwards, I deleted all 100 work chains, and then restarted them with a 5 second pause in between. Now all work chains were able to create the PwCalculations without issue.
I'm running:
OS: Ubuntu 18.04.5 LTS
Python: 3.8
aiida-core: 1.6.3 (but I think I've also spotted this issue when I was running v1.6.1, just couldn't pin it down at the time)
plumpy: 0.19.0
kiwipy: 0.7.4
sqlalchemy: 1.3.23
Seeing the same thing, also when submitting a large amount of workchains (whether one-by-one in a script or by a parent workchain launching them does not matter, but when launching them via a parent workchain the issue is triggered sooner).
The backtrace is a bit different, though:
2021-05-14 10:55:09 [1739 | REPORT]: [6047|Cp2kEosWorkChain|on_except]: Traceback (most recent call last):
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/process_states.py", line 230, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/scratch/tiziano/work/aiida/aiida-cp2k/aiida_cp2k/workchains/eos.py", line 170, in run_calculations
builder.cp2k.structure = get_rescaled_structure(self.inputs.structure, Float(scale))
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/functions.py", line 179, in decorated_function
result, _ = run_get_node(*args, **kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/functions.py", line 131, in run_get_node
process = process_class(inputs=inputs, runner=runner)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 380, in on_entering
super().on_entering(state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 376, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 563, in _create_and_setup_db_record
self._setup_db_record()
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/functions.py", line 362, in _setup_db_record
super()._setup_db_record()
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 672, in _setup_db_record
self._setup_inputs()
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/node.py", line 417, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/utils/mixins.py", line 148, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/node.py", line 449, in validate_incoming
if builder.count() > 0:
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/querybuilder.py", line 2164, in count
return self._impl.count(query)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/implementation/querybuilder.py", line 288, in count
return query.count()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3803, in count
return self.from_self(col).scalar()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3523, in scalar
ret = self.one()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1115, in _execute_clauseelement
compiled_sql = elem.compile(
File "<string>", line 1, in <lambda>
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1801, in visit_alias
ret = alias.original._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2435, in visit_join
join.left._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2437, in visit_join
+ join.right._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1727, in visit_cte
self.visit_cte(pre_alias_cte, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1758, in visit_cte
cte.original._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1216, in visit_compound_select
text = (" " + keyword + " ").join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1218, in <genexpr>
c._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2301, in _compose_select_body
t = select._whereclause._compiler_dispatch(self, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1354, in visit_binary
return disp(binary, operator_, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1471, in visit_like_op_binary
binary.left._compiler_dispatch(self, **kw),
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1075, in visit_cast
cast.clause._compiler_dispatch(self, **kwargs),
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/annotation.py", line 79, in _compiler_dispatch
return self.__element.__class__._compiler_dispatch(self, visitor, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 912, in visit_column
if not is_literal and isinstance(name, elements._truncated_label):
RecursionError: maximum recursion depth exceeded while calling a Python object
aiida-core: 403f7e7d896f8408d0cacee9fe41c030a1072eaf
plumpy: 0.19.0
kiwipy: 0.7.4
sqlalchemy: 1.3.24
python: 3.9.5
Just want to say that I did a similar thing like submitting the work chains (~100 VaspRelaxWorkChain) in a loop without delay previously (aiida-core < 1.6.1), and found no issue. In my case, the submission process itself is rather slow though, takes about 1.5 seconds for each call to submit.
And another one, again at a different place (but this one comes from the sub-workchain called by the primary one):
$ verdi process report 15370
2021-05-17 10:26:29 [7359 | REPORT]: [15370|Cp2kBaseWorkChain|on_except]: Traceback (most recent call last):
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/process_states.py", line 230, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 532, in step
finished, result = self._child_stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/workchains/restart.py", line 183, in run_process
node = self.submit(self.process_class, **inputs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 498, in submit
return self.runner.submit(process, *args, **kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/runners.py", line 184, in submit
process_inited = self.instantiate_process(process, *args, **inputs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/runners.py", line 170, in instantiate_process
return instantiate_process(self, process, *args, **inputs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/utils.py", line 65, in instantiate_process
process = process_class(runner=runner, inputs=inputs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 380, in on_entering
super().on_entering(state)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 376, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 563, in _create_and_setup_db_record
self._setup_db_record()
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 672, in _setup_db_record
self._setup_inputs()
File "/scratch/tiziano/work/aiida/aiida-core/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/node.py", line 417, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/utils/mixins.py", line 148, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/nodes/node.py", line 449, in validate_incoming
if builder.count() > 0:
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/querybuilder.py", line 2164, in count
return self._impl.count(query)
File "/scratch/tiziano/work/aiida/aiida-core/aiida/orm/implementation/querybuilder.py", line 288, in count
return query.count()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3803, in count
return self.from_self(col).scalar()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3523, in scalar
ret = self.one()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1115, in _execute_clauseelement
compiled_sql = elem.compile(
File "<string>", line 1, in <lambda>
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1801, in visit_alias
ret = alias.original._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2435, in visit_join
join.left._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2437, in visit_join
+ join.right._compiler_dispatch(self, asfrom=True, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1727, in visit_cte
self.visit_cte(pre_alias_cte, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1758, in visit_cte
cte.original._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1216, in visit_compound_select
text = (" " + keyword + " ").join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1218, in <genexpr>
c._compiler_dispatch(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 2301, in _compose_select_body
t = select._whereclause._compiler_dispatch(self, **kwargs)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1354, in visit_binary
return disp(binary, operator_, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1471, in visit_like_op_binary
binary.left._compiler_dispatch(self, **kw),
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/scratch/tiziano/virtualenvs/aiida/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1075, in visit_cast
cast.clause._compiler_dispatch(self, **kwargs),
RecursionError: maximum recursion depth exceeded in comparison
I took the liberty to add the important label, because the mess this leaves is considerable (one also has to check all subprocesses).
@danieleongari is the computer to run the calculations configured with an SSH proxy command?
Just to report the same issue, submitting many common workflows. I have two different behaviours (probably because of the different place where the recursion limit occurs):
2021-06-11 07:03:21 [1133 | REPORT]: [6664|EquationOfStateWorkChain|run_init]: submitting `QuantumEspressoCommonRelaxWorkChain` for scale_factor `uuid: 8c5be874-7345-478e-9139-e44f3a6e00cc (pk: 6758) value: 0.94`
2021-06-11 07:04:02 [1193 | REPORT]: [7525|PwRelaxWorkChain|setup]: No change in volume possible for the provided base input parameters. Meta convergence is turned off.
2021-06-11 07:04:02 [1194 | REPORT]: [7525|PwRelaxWorkChain|setup]: Work chain will not run final SCF when `calculation` is set to `scf` for the relaxation `PwBaseWorkChain`.
2021-06-11 07:04:03 [1199 | REPORT]: [7525|PwRelaxWorkChain|run_relax]: launching PwBaseWorkChain<7628>
2021-06-11 07:05:01 [1451 | REPORT]: [7628|PwBaseWorkChain|run_process]: launching PwCalculation<9311> iteration #1
2021-06-11 07:48:44 [4902 | REPORT]: [7628|PwBaseWorkChain|results]: work chain completed after 1 iterations
2021-06-11 07:48:44 [4903 | REPORT]: [7628|PwBaseWorkChain|on_terminated]: remote folders will not be cleaned
2021-06-11 07:49:26 [4924 | REPORT]: [7525|PwRelaxWorkChain|results]: workchain completed after 1 iterations
2021-06-11 07:49:29 [4928 | REPORT]: [7525|PwRelaxWorkChain|on_terminated]: cleaned remote folders of calculations: 9311
2021-06-11 07:50:03 [4960 | REPORT]: [6664|EquationOfStateWorkChain|on_except]: Traceback (most recent call last):
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/process_states.py", line 230, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-common-workflows/aiida_common_workflows/workflows/eos.py", line 196, in run_eos
builder, structure = self.get_sub_workchain_builder(scale_factor, reference_workchain=reference_workchain)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-common-workflows/aiida_common_workflows/workflows/eos.py", line 160, in get_sub_workchain_builder
structure = scale_structure(self.inputs.structure, scale_factor)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 179, in decorated_function
result, _ = run_get_node(*args, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 131, in run_get_node
process = process_class(inputs=inputs, runner=runner)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 380, in on_entering
super().on_entering(state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 376, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 563, in _create_and_setup_db_record
self._setup_db_record()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 362, in _setup_db_record
super()._setup_db_record()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 672, in _setup_db_record
self._setup_inputs()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/node.py", line 802, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/utils/mixins.py", line 139, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/node.py", line 834, in validate_incoming
if builder.count() > 0:
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/querybuilder.py", line 2193, in count
return self._impl.count(query)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/implementation/querybuilder.py", line 290, in count
return query.count()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3803, in count
return self.from_self(col).scalar()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3523, in scalar
ret = self.one()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1121, in _execute_clauseelement
else None,
File "<string>", line 1, in <lambda>
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in _compose_select_body
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in <listcomp>
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1802, in visit_alias
self, asfrom=True, **kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in _compose_select_body
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in <listcomp>
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2439, in visit_join
+ join.onclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2439, in visit_join
+ join.onclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1727, in visit_cte
self.visit_cte(pre_alias_cte, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1759, in visit_cte
self, asfrom=True, **kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1225, in visit_compound_select
for i, c in enumerate(cs.selects)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1225, in <genexpr>
for i, c in enumerate(cs.selects)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2301, in _compose_select_body
t = select._whereclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in visit_clauselist
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1041, in <genexpr>
s
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in visit_clauselist
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1041, in <genexpr>
s
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1354, in visit_binary
return disp(binary, operator_, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1471, in visit_like_op_binary
binary.left._compiler_dispatch(self, **kw),
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1076, in visit_cast
cast.typeclause._compiler_dispatch(self, **kwargs),
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 955, in visit_typeclause
return self.dialect.type_compiler.process(typeclause.type, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 402, in process
return type_._compiler_dispatch(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 89, in _compiler_dispatch
meth = getter(visitor)
RecursionError: maximum recursion depth exceeded while calling a Python object
2021-06-11 07:04:01 [1190 | REPORT]: [6733|EquationOfStateWorkChain|run_init]: submitting `QuantumEspressoCommonRelaxWorkChain` for scale_factor `uuid: 41b191e3-c9f5-4bb1-b038-0099d99123eb (pk: 6963) value: 0.94`
2021-06-11 07:04:21 [1296 | REPORT]: [8136|PwRelaxWorkChain|setup]: No change in volume possible for the provided base input parameters. Meta convergence is turned off.
2021-06-11 07:04:21 [1297 | REPORT]: [8136|PwRelaxWorkChain|setup]: Work chain will not run final SCF when `calculation` is set to `scf` for the relaxation `PwBaseWorkChain`.
2021-06-11 07:04:22 [1301 | REPORT]: [8136|PwRelaxWorkChain|run_relax]: launching PwBaseWorkChain<8252>
2021-06-11 07:04:28 [1335 | REPORT]: [8252|PwBaseWorkChain|run_process]: launching PwCalculation<8449> iteration #1
2021-06-11 07:49:38 [4940 | REPORT]: [8252|PwBaseWorkChain|results]: work chain completed after 1 iterations
2021-06-11 07:49:39 [4941 | REPORT]: [8252|PwBaseWorkChain|on_terminated]: remote folders will not be cleaned
2021-06-11 07:49:52 [4950 | REPORT]: [8136|PwRelaxWorkChain|results]: workchain completed after 1 iterations
2021-06-11 07:49:56 [4951 | REPORT]: [8136|PwRelaxWorkChain|on_terminated]: cleaned remote folders of calculations: 8449
2021-06-11 07:50:01 [4955 | REPORT]: [7611|QuantumEspressoCommonRelaxWorkChain|on_except]: Traceback (most recent call last):
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/process_states.py", line 230, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-common-workflows/aiida_common_workflows/workflows/relax/quantum_espresso/workchain.py", line 49, in convert_outputs
result = extract_from_parameters(outputs.output_parameters).values()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 179, in decorated_function
result, _ = run_get_node(*args, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 131, in run_get_node
process = process_class(inputs=inputs, runner=runner)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 380, in on_entering
super().on_entering(state)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 376, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 563, in _create_and_setup_db_record
self._setup_db_record()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/functions.py", line 362, in _setup_db_record
super()._setup_db_record()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 672, in _setup_db_record
self._setup_inputs()
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/node.py", line 802, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/utils/mixins.py", line 139, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/nodes/node.py", line 834, in validate_incoming
if builder.count() > 0:
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/querybuilder.py", line 2193, in count
return self._impl.count(query)
File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-core/aiida/orm/implementation/querybuilder.py", line 290, in count
return query.count()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3803, in count
return self.from_self(col).scalar()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3523, in scalar
ret = self.one()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1121, in _execute_clauseelement
else None,
File "<string>", line 1, in <lambda>
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in _compose_select_body
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in <listcomp>
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1802, in visit_alias
self, asfrom=True, **kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in _compose_select_body
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2294, in <listcomp>
for f in froms
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2439, in visit_join
+ join.onclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2439, in visit_join
+ join.onclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1727, in visit_cte
self.visit_cte(pre_alias_cte, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1759, in visit_cte
self, asfrom=True, **kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1225, in visit_compound_select
for i, c in enumerate(cs.selects)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1225, in <genexpr>
for i, c in enumerate(cs.selects)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2202, in visit_select
text, select, inner_columns, froms, byfrom, kwargs
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 2301, in _compose_select_body
t = select._whereclause._compiler_dispatch(self, **kwargs)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in visit_clauselist
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1041, in <genexpr>
s
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in visit_clauselist
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1041, in <genexpr>
s
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1354, in visit_binary
return disp(binary, operator_, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1471, in visit_like_op_binary
binary.left._compiler_dispatch(self, **kw),
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 1076, in visit_cast
cast.typeclause._compiler_dispatch(self, **kwargs),
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 955, in visit_typeclause
return self.dialect.type_compiler.process(typeclause.type, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 402, in process
return type_._compiler_dispatch(self, **kw)
File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
RecursionError: maximum recursion depth exceeded in comparison
2021-06-11 07:50:57 [5002 | REPORT]: [6733|EquationOfStateWorkChain|inspect_init]: Initial sub process did not finish successful so aborting the workchain.
The validate_incoming function of a work function seems to be (?) a common denominator of many of these reports (also of others?)
@chrisjsewell can you please look into this? This seems like a blocker and important issue...
@chrisjsewell can you please look into this? This seems like a blocker and important issue...
Indeed š Iām going to do a bunch of aiida-core stuff next week
The validate_incoming function of a work function seems to be (?)
Not sure. Looking at the original example in the OP happens when a Process gets deserialized from a process checkpoint. The examples by @dev-zero are in one case a calcfunction that is called, but the other example is the submission of a process. Both those cases point to the instantiation of a process, but are still different from the original case which excepts while loading a simple node from the database.
BTW:
- I confirm that I'm using SSH with a proxy_command; looking in the log, sometimes (but not always) the recursion error seems to be associated with transport issues
- I note that I was running with 8 workers, and I increased the number of slots
daemon.worker_process_slotsto 1200 - I have restarted a few times the daemon today (but I cannot say if the errors are correlated or not with the daemon shutdowns)
- in a few cases, there seem to be a very long loop of recursive calls to exceptions like this, maybe this is (partially) related?
Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 265, in validate_kpoints kpoints = self.inputs.kpoints File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 131, in __getattr__ raise AttributeError(errmsg) AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 128, in __getattr__ return self[attr] File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 85, in __getitem__ return self._dict[key] KeyError: 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 265, in validate_kpoints kpoints = self.inputs.kpoints File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 131, in __getattr__ raise AttributeError(errmsg) AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 128, in __getattr__ return self[attr] File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 85, in __getitem__ return self._dict[key] KeyError: 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 265, in validate_kpoints kpoints = self.inputs.kpoints File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 131, in __getattr__ raise AttributeError(errmsg) AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 128, in __getattr__ return self[attr] File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 85, in __getitem__ return self._dict[key] KeyError: 'kpoints' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pizzi/.virtualenvs/aiida-prod/codes/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 265, in validate_kpoints kpoints = self.inputs.kpoints File "/home/pizzi/.virtualenvs/aiida-prod/lib/python3.7/site-packages/plumpy/utils.py", line 131, in __getattr__ raise AttributeError(errmsg) AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints' During handling of the above exception, another exception occurred: ... - I continue to see these crashes even much after the original submission (hours later), they are quite negatively impacting the results (of the work chains that finished, 217 finished OK and 141 failed, and from a quick look the vast majority because of this bug)
Since I don't think there is any sensitive information, I attach the full log (I started running from a clean profile yesterday, so the log should contain all info, not cluttered with too much else): daemon.log.zip
When performing FLEUR scandium relaxation I ran into this error which Vasily described yesterday in the mailing list. Here you find a full traceback:
2021-08-10 13:44:18 [34571 | REPORT]: [55184|FleurCreateMagneticWorkChain|start]: INFO: started Create Magnetic Film workflow version 0.2.0
2021-08-10 13:44:18 [34572 | REPORT]: [55184|FleurCreateMagneticWorkChain|start]: INFO: EOS workchain will be submitted
2021-08-10 13:44:18 [34573 | REPORT]: [55184|FleurCreateMagneticWorkChain|run_eos]: INFO: submit EOS WorkChain
2021-08-10 13:44:18 [34574 | REPORT]: [55184|FleurCreateMagneticWorkChain|on_except]: Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/plumpy/process_states.py", line 230, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/opt/aiida-core/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/usr/local/lib/python3.8/dist-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/usr/local/lib/python3.8/dist-packages/plumpy/workchains.py", line 432, in step
finished, retval = self._child_stepper.step()
File "/usr/local/lib/python3.8/dist-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/usr/local/lib/python3.8/dist-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/opt/aiida-fleur/aiida_fleur/workflows/create_magnetic_film.py", line 124, in run_eos
inputs, error = self.prepare_eos()
File "/opt/aiida-fleur/aiida_fleur/workflows/create_magnetic_film.py", line 111, in prepare_eos
inputs.structure = create_substrate_bulk(Dict(dict=self.ctx.wf_dict))
File "/opt/aiida-core/aiida/engine/processes/functions.py", line 179, in decorated_function
result, _ = run_get_node(*args, **kwargs)
File "/opt/aiida-core/aiida/engine/processes/functions.py", line 131, in run_get_node
process = process_class(inputs=inputs, runner=runner)
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/usr/local/lib/python3.8/dist-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/opt/aiida-core/aiida/engine/processes/process.py", line 380, in on_entering
super().on_entering(state)
File "/usr/local/lib/python3.8/dist-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/usr/local/lib/python3.8/dist-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/opt/aiida-core/aiida/engine/processes/process.py", line 376, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/opt/aiida-core/aiida/engine/processes/process.py", line 563, in _create_and_setup_db_record
self._setup_db_record()
File "/opt/aiida-core/aiida/engine/processes/functions.py", line 362, in _setup_db_record
super()._setup_db_record()
File "/opt/aiida-core/aiida/engine/processes/process.py", line 672, in _setup_db_record
self._setup_inputs()
File "/opt/aiida-core/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/opt/aiida-core/aiida/orm/nodes/node.py", line 802, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/opt/aiida-core/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/opt/aiida-core/aiida/orm/utils/mixins.py", line 139, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/opt/aiida-core/aiida/orm/nodes/node.py", line 834, in validate_incoming
if builder.count() > 0:
File "/opt/aiida-core/aiida/orm/querybuilder.py", line 2193, in count
return self._impl.count(query)
File "/opt/aiida-core/aiida/orm/implementation/querybuilder.py", line 290, in count
return query.count()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3803, in count
return self.from_self(col).scalar()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3523, in scalar
ret = self.one()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3490, in one
ret = self.one_or_none()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
ret = list(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1115, in _execute_clauseelement
compiled_sql = elem.compile(
File "<string>", line 1, in <lambda>
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 481, in compile
return self._compiler(dialect, bind=bind, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 487, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 592, in __init__
Compiled.__init__(self, dialect, statement, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 322, in __init__
self.string = self.process(self.statement, **compile_kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 352, in process
return obj._compiler_dispatch(self, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1801, in visit_alias
ret = alias.original._compiler_dispatch(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2292, in _compose_select_body
[
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2293, in <listcomp>
f._compiler_dispatch(self, asfrom=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2435, in visit_join
join.left._compiler_dispatch(self, asfrom=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2437, in visit_join
+ join.right._compiler_dispatch(self, asfrom=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1727, in visit_cte
self.visit_cte(pre_alias_cte, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1758, in visit_cte
cte.original._compiler_dispatch(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1216, in visit_compound_select
text = (" " + keyword + " ").join(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1218, in <genexpr>
c._compiler_dispatch(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2201, in visit_select
text = self._compose_select_body(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 2301, in _compose_select_body
t = select._whereclause._compiler_dispatch(self, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1040, in visit_clauselist
text = sep.join(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1040, in <genexpr>
text = sep.join(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1043, in <genexpr>
c._compiler_dispatch(self, **kw) for c in clauselist.clauses
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1354, in visit_binary
return disp(binary, operator_, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1471, in visit_like_op_binary
binary.left._compiler_dispatch(self, **kw),
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/compiler.py", line 1075, in visit_cast
cast.clause._compiler_dispatch(self, **kwargs),
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/annotation.py", line 79, in _compiler_dispatch
return self.__element.__class__._compiler_dispatch(self, visitor, **kw)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/visitors.py", line 89, in _compiler_dispatch
meth = getter(visitor)
RecursionError: maximum recursion depth exceeded while calling a Python object
FleurCreateMagneticWorkChain<55184> Excepted [1:if_(eos_needed)]
This and similar tracebacks occur on various points of the work chain.
Thanks, I think this error is maybe different to the one opened for this issue. For this one, at least for initial debugging, I would think to maybe add to: https://github.com/aiidateam/aiida-core/blob/4174e5de3adbeec785290a02a0fc78d4597e42e0/aiida/orm/nodes/node.py#L455-L456
something like:
try:
count = builder.count()
except Exception as exc:
raise ValueError(f'the link ({source} -> {self}) would result in an erroneous query') from exc
if count > 0:
raise ValueError(f'the link you are attempting to create ({source} -> {self}) would generate a cycle in the graph')
at least then we could see what nodes are trying to be linked
I got a chance to reproduce this issue and I add the exception trace as @chrisjsewell suggested. Here is the traces I got. It is basically lots of kpoints KeyError followed with RecursionError describe in this issue:
Traceback (most recent call last):
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/utils.py", line 128, in __getattr__
return self[attr]
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/utils.py", line 85, in __getitem__
return self._dict[key]
KeyError: 'kpoints'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jyu/Projects/WP-SSSP/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 273, in validate_kpoints
kpoints = self.inputs.kpoints
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/utils.py", line 131, in __getattr__
raise AttributeError(errmsg)
AttributeError: 'AttributesFrozendict' object has no attribute 'kpoints'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/nodes/node.py", line 835, in validate_incoming
count = builder.count()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/querybuilder.py", line 2193, in count
return self._impl.count(query)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/implementation/querybuilder.py", line 290, in count
return query.count()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3803, in count
...
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 96, in _compiler_dispatch
return meth(self, **kw)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1075, in visit_cast
cast.clause._compiler_dispatch(self, **kwargs),
RecursionError: maximum recursion depth exceeded in comparison
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/process_states.py", line 231, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/engine/processes/workchains/workchain.py", line 214, in _do_step
finished, stepper_result = self._stepper.step()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/workchains.py", line 532, in step
finished, result = self._child_stepper.step()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
...
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/engine/processes/process.py", line 709, in _setup_inputs
self.node.add_incoming(node, LinkType.INPUT_CALC, name)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/nodes/node.py", line 802, in add_incoming
self.validate_incoming(source, link_type, link_label)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/nodes/process/process.py", line 472, in validate_incoming
super().validate_incoming(source, link_type, link_label)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/utils/mixins.py", line 139, in validate_incoming
super().validate_incoming(source, link_type=link_type, link_label=link_label)
File "/home/jyu/miniconda3/envs/aiida-sssp-dev/lib/python3.9/site-packages/aiida/orm/nodes/node.py", line 837, in validate_incoming
raise ValueError(f'the link ({source} -> {self}) would result in an erroneous query') from exc
ValueError: the link (Remote code 'pw-6.8' on eiger-hq, pk: 1, uuid: dc38cb79-defb-4fc9-a859-58e44aabefe7 -> uuid: 799e0295-a8b7-4c81-98b1-5511c87d0c21 (unstored) (aiida.calculations:quantumespresso.pw)) would result in an erroneous query
Please let me know if you have any idea on how to further debug this @chrisjsewell
Can the people that reported a case of this please indicate whether they were using the Django or SqlAlchemy backend? I wonder if this has something to do with the SqlAlchemy engine being created for the QueryBuilder in the Django backend and not being reset correctly. So I wonder if this mostly occurs for Django backends, and not the SqlAlchemy ones. That would be a great hint.
Got another hit of this, but on develop (commit 09765ecbd8280b629da5a804bdaa6353fc0a3179) with an SqlAlchemy backend (is the only remaining). Just running roughly 300 processes in parallel. Will try to see if it happens again when lowering the amount of concurrent processes.
Traceback (most recent call last):
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/kiwipy/rmq/tasks.py", line 166, in _on_task
result = await result
File "/usr/lib/python3.9/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
future.result()
File "/usr/lib/python3.9/asyncio/futures.py", line 201, in result
raise self._exception
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/kiwipy/rmq/threadcomms.py", line 284, in done
result = kiwi_future.result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/kiwipy/futures.py", line 54, in capture_exceptions
yield
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/communications.py", line 48, in on_done
result = plum_future.result()
File "/usr/lib/python3.9/asyncio/futures.py", line 201, in result
raise self._exception
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/kiwipy/futures.py", line 54, in capture_exceptions
yield
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/futures.py", line 73, in run_task
res = await coro()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/process_comms.py", line 539, in __call__
return await self._continue(communicator, **task.get(TASK_ARGS, {}))
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/manage/external/rmq.py", line 208, in _continue
result = await super()._continue(communicator, pid, nowait, tag)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/process_comms.py", line 615, in _continue
return proc.future().result()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/process_states.py", line 231, in execute
result = self.run_fn(*self.args, **self.kwargs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/processes/workchains/workchain.py", line 252, in _do_step
finished, stepper_result = self._stepper.step()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/workchains.py", line 299, in step
finished, result = self._child_stepper.step()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/workchains.py", line 250, in step
return True, self._fn(self._workchain)
File "/home/sph/code/aiida/env/dev/aiida-codtools/aiida_codtools/workflows/cif_clean.py", line 103, in run_select_calculation
calculation = self.submit(CifSelectCalculation, **inputs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/processes/process.py", line 509, in submit
return self.runner.submit(process, *args, **kwargs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/runners.py", line 183, in submit
process_inited = self.instantiate_process(process, *args, **inputs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/runners.py", line 169, in instantiate_process
return instantiate_process(self, process, *args, **inputs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/utils.py", line 65, in instantiate_process
process = process_class(runner=runner, inputs=inputs)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 193, in __call__
inst.transition_to(inst.create_initial_state())
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 335, in transition_to
self.transition_failed(initial_state_label, label, *sys.exc_info()[1:])
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 351, in transition_failed
raise exception.with_traceback(trace)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 320, in transition_to
self._enter_next_state(new_state)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 382, in _enter_next_state
self._fire_state_event(StateEventHook.ENTERING_STATE, next_state)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/state_machine.py", line 299, in _fire_state_event
callback(self, hook, state)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/processes.py", line 324, in <lambda>
lambda _s, _h, state: self.on_entering(cast(process_states.State, state)),
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/processes/process.py", line 391, in on_entering
super().on_entering(state)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/processes.py", line 669, in on_entering
call_with_super_check(self.on_create)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/processes/process.py", line 387, in on_create
self._pid = self._create_and_setup_db_record() # pylint: disable=attribute-defined-outside-init
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/engine/processes/process.py", line 577, in _create_and_setup_db_record
self.node.store_all()
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 678, in store_all
return self.store(with_transaction)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 715, in store
self._store(with_transaction=with_transaction, clean=True)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 746, in _store
self._backend_entity.set_extra(self._HASH_EXTRA_KEY, self.get_hash())
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 811, in get_hash
return self._get_hash(ignore_errors=ignore_errors, **kwargs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 822, in _get_hash
return make_hash(self._get_objects_to_hash(), **kwargs)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/process/calculation/calcjob.py", line 134, in _get_objects_to_hash
for entry in self.get_incoming(link_type=(LinkType.INPUT_CALC, LinkType.INPUT_WORK))
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 604, in get_incoming
link_triples = self.get_stored_link_triples(
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/nodes/node.py", line 579, in get_stored_link_triples
return [LinkTriple(entry[0], LinkType(entry[1]), entry[2]) for entry in builder.all()]
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/querybuilder.py", line 1071, in all
matches = list(self.iterall(batch_size=batch_size))
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/orm/querybuilder.py", line 1033, in iterall
for item in self._impl.iterall(self.as_dict(), batch_size):
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/storage/psql_dos/orm/querybuilder/main.py", line 172, in iterall
with self.use_query(data) as query:
File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/storage/psql_dos/orm/querybuilder/main.py", line 204, in use_query
query = self._update_query(data)
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/storage/psql_dos/orm/querybuilder/main.py", line 226, in _update_query
self._build()
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/storage/psql_dos/orm/querybuilder/main.py", line 292, in _build
result = join_func(
File "/home/sph/code/aiida/env/dev/aiida-core/aiida/storage/psql_dos/orm/querybuilder/joiner.py", line 349, in _join_node_inputs
).join(entity_to_join, aliased_edge.input_id == entity_to_join.id, isouter=isouterjoin)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/operators.py", line 360, in __eq__
return self.operate(eq, other)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 317, in operate
return op(self.comparator, *other, **kwargs)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/operators.py", line 360, in __eq__
return self.operate(eq, other)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/properties.py", line 431, in operate
return op(self.__clause_element__(), *other, **kwargs)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/annotation.py", line 221, in __eq__
return self.__element.__class__.__eq__(self, other)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/operators.py", line 360, in __eq__
return self.operate(eq, other)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 861, in operate
return op(self.comparator, *other, **kwargs)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/operators.py", line 360, in __eq__
return self.operate(eq, other)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/type_api.py", line 76, in operate
return o[0](self.expr, op, *(other + o[1:]), **kwargs)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/default_comparator.py", line 101, in _boolean_compare
obj = coercions.expect(
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/coercions.py", line 172, in expect
element = element.__clause_element__()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 259, in __clause_element__
return self.expression
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 235, in expression
ce = self.comparator.__clause_element__()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1227, in oneshot
result = fn(*args, **kw)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/properties.py", line 393, in _memoized_method___clause_element__
return self.adapter(self.prop.columns[0], self.prop.key)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/orm/util.py", line 813, in _adapt_element
self._adapter.traverse(elem)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/util.py", line 998, in traverse
return self.columns[obj]
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/util/_collections.py", line 762, in __missing__
self[key] = val = self.creator(self.weakself(), key)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/util.py", line 1024, in _locate_col
c = vis.replace(col, _include_singleton_constants=True)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/util.py", line 908, in replace
return self._corresponding_column(col, True)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/util.py", line 833, in _corresponding_column
newcol = self.selectable.corresponding_column(
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 226, in corresponding_column
return self.exported_columns.corresponding_column(
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 718, in exported_columns
return self.columns
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 737, in columns
self._populate_column_collection()
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 1643, in _populate_column_collection
self.element._generate_fromclause_column_proxies(self)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 694, in _generate_fromclause_column_proxies
fromclause._columns._populate_separate_keys(
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1293, in _populate_separate_keys
cols = list(iter_)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 695, in <genexpr>
col._make_proxy(fromclause) for col in self.c
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 1969, in _make_proxy
c = self._constructor(
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 1670, in __init__
self._init_items(*args)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 135, in _init_items
spwd(self, **kw)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 1046, in _set_parent_with_dispatch
self._set_parent(parent, **kw)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 2445, in _set_parent
self.parent._on_table_attach(self._set_table)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 1879, in _on_table_attach
event.listen(self, "after_parent_attach", fn)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/event/api.py", line 115, in listen
_event_key(target, identifier, fn).listen(*args, **kw)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/event/api.py", line 25, in _event_key
tgt = evt_cls._accept_with(target)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/event/base.py", line 245, in _accept_with
if hasattr(target, "dispatch"):
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/event/base.py", line 321, in __get__
disp = self.dispatch._for_instance(obj)
File "/home/sph/.virtualenvs/aiida_dev/lib/python3.9/site-packages/sqlalchemy/event/base.py", line 127, in _for_instance
return self._for_class(instance_cls)
RecursionError: maximum recursion depth exceeded
Note that when running with 30 processes in parallel (single daemon worker) I no longer experienced any problems. Even after having run roughly 45k processes. So I have the feeling the problem appears only when under heavy load. Not sure if it is necessary to have multiple daemon workers, or simply have many processes active in one worker. Will try to run many processes in parallel with just a single worker.
Can confirm that I reproduced the problem with just a single daemon worker under heavy load. So it doesn't seem to require multiple active workers.
Can confirm that I reproduced the problem with just a single daemon worker under heavy load. So it doesn't seem to require multiple active workers.
I also just ran into this (aiida-core== 1.6.8, sqlalchemy==1.3.24, kiwipy== 0.7.5, Python=3.9.12 on an Ubuntu 20 server) and, at least in my case, it seemed to be correlated to also running out of slots.
