Async runtime fails to go back to sane state after exception

[erick-xanadu]

Issue description

• Expected behavior: After executing a qnode which raises an exception, it is expected that executing a second qnode will have a fresh environment.

• Actual behavior: Something is not being reset correctly when raising an exception. This can be easily verified by running the async tests for exceptions before running the async tests that succeed. If the order changes, some tests which are expected to succeed will raise an error.

• Reproduces how often: 100%

import pennylane as qml

@qml.qjit(async_qnodes=True)
@qml.qnode(qml.device("lightning.qubit", wires=2))
def foo(a, b):
  qml.CNOT(wires=[a, b])
  return qml.state()

print(foo(0, 1)) # Different wires, async shouldn't happen.

try:
  foo(0, 0) # Same wires, will result in an exception
except:
  ...


print(foo(0, 1)) # Different wires, async shouldn't happen.