ProcessNode "hides" that it is cached

[ltalirz]

Describe the bug

I recently encountered a case where a ProcessNode clearly appeared cached (ctime, mtime within fraction of a second; connected calculation directory had files touched before ctime of the node), but node.base.caching.get_cache_source() came back empty.

After investigating with @sphuber we figured out that

1. The node it was cached from (determined by looking up the UUID in .aiida/calcinfo.json of the attached calculation folder) had gotten into a weird state, where its node.extras['_aiida_hash'] did not match the result of node._get_hash().
2. The cached node was missing the _aiida_cached_from extra

Other relevant context:

• the attributes of the two nodes were identical, except that the source node had an 'sealed': True attribute that was missing from the cached node. In particular, the version attribute that tracks core (2.0.4.post0) and plugin versions was identical.
• I did not update aiida-core / the plugin (aiida-cusp) in between the two runs

Steps to reproduce

Unfortunately, I do not (yet) know how to reproduce this.

If anyone has an idea of what could cause this, please comment.

Expected behavior

• node.extras['_aiida_hash'] and node._get_hash() should match for the source node
• the cached node should have the appropriate _aiida_cached_from extra

Your environment

• Operating system [e.g. Linux]: ubuntu 20.04
• Python version [e.g. 3.7.1]: 3.9
• aiida-core version [e.g. 1.2.1]: current main

Other relevant software versions, e.g. Postres & RabbitMQ

Additional context