aiidateam
Fix FileNotFoundError in monitors and importers using improper NamedTemporaryFile pattern
The always_kill monitor and ArithmeticAddCalculationImporter were causing FileNotFoundError when transport.getfile() attempted to write to temporary files created with NamedTemporaryFile('w+'). The open file handle conflicted with external file access, particularly on Windows and certain filesystems.
Changes
-
Replace
NamedTemporaryFile(delete=True)withtempfile.mkstemp()in monitor and importer code- Matches existing pattern in
cmd_computer.pyfor transport operations - Explicitly close file descriptor before external access
- Manual cleanup in
finallyblock with OSError handling
- Matches existing pattern in
-
Add minimal tests to verify temporary file cleanup
-
test_always_kill_tempfile_cleanup: Verifies the monitor properly cleans up temporary files -
test_parse_remote_data_tempfile_cleanup: Verifies the importer properly cleans up temporary files
-
Before:
with tempfile.NamedTemporaryFile('w+') as handle:
cwd = node.get_remote_workdir()
transport.getfile(str(Path(cwd).joinpath('_aiidasubmit.sh')), handle.name)
handle.seek(0)
output = handle.read()
After:
handle, temp_path = tempfile.mkstemp(suffix='.sh')
os.close(handle)
try:
transport.getfile(str(Path(cwd).joinpath('_aiidasubmit.sh')), temp_path)
with open(temp_path, 'r', encoding='utf-8') as f:
output = f.read()
finally:
try:
os.remove(temp_path)
except OSError:
pass
Files Modified
-
src/aiida/calculations/monitors/base.py-always_kill()function -
src/aiida/calculations/importers/arithmetic/add.py-ArithmeticAddCalculationImporter.parse_remote_data()method -
tests/engine/processes/calcjobs/test_monitors.py- Added cleanup test foralways_kill() -
tests/calculations/importers/arithmetic/test_add.py- Added cleanup test forparse_remote_data()
Original prompt
This section details on the original issue you should resolve
<issue_title>🐛 FileNotFoundError related to monitors</issue_title>
<issue_description>While "running" a workflow, I am reliably running into the following error:
FileNotFoundError: [Errno 2] No such file
From the full traceback (see below), the issue seems to be related to the monitors. In the end, the workflow does complete successfully, but the issue produces a lot of noise that will worry users.
I tested both core.ssh and core.ssh_async, and am running into the issue for both.
Full Traceback
11/05/2025 11:52:01 AM <1716> aiida.engine.transports: [ERROR] Exception whilst using transport:
Traceback (most recent call last):
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/transports.py", line 106, in request_transport
yield transport_request.future
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/tasks.py", line 257, in do_monitor
return monitors.process(node, transport)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/monitors.py", line 204, in process
monitor_result = monitor_function(node, transport, **monitor.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-vasp/src/aiida_vasp/calcs/monitors.py", line 66, in monitor_stdout
file_stat = transport.get_attribute(stdout_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 1201, in get_attribute
paramiko_attr = self.lstat(path)
^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 662, in lstat
return self.sftp.lstat(path)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 511, in lstat
t, msg = self._request(CMD_LSTAT, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 857, in _request
return self._read_response(num)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 909, in _read_response
self._convert_status(msg)
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
11/05/2025 11:52:01 AM <1716> aiida.orm.nodes.process.calculation.calcjob.CalcJobNode: [ERROR] iteration 1 of do_monitor excepted, retrying after 20 seconds
Traceback (most recent call last):
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/utils.py", line 205, in exponential_backoff_retry
result = await coro()
^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/tasks.py", line 257, in do_monitor
return monitors.process(node, transport)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/engine/processes/calcjobs/monitors.py", line 204, in process
monitor_result = monitor_function(node, transport, **monitor.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-vasp/src/aiida_vasp/calcs/monitors.py", line 66, in monitor_stdout
file_stat = transport.get_attribute(stdout_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 1201, in get_attribute
paramiko_attr = self.lstat(path)
^^^^^^^^^^^^^^^^
File "/Users/mbercx/project/defect/git/aiida-core/src/aiida/transports/plugins/ssh.py", line 662, in lstat
return self.sftp.lstat(path)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 511, in lstat
t, msg = self._request(CMD_LSTAT, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 857, in _request
return self._read_response(num)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 909, in _read_response
self._convert_status(msg)
File "/Users/mbercx/.aiida_venvs/defect/lib/python3.12/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
Comments on the Issue (you ar...
- Fixes aiidateam/aiida-core#7086
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
Codecov Report
:x: Patch coverage is 42.30769% with 15 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.61%. Comparing base (396792b) to head (402d057).
:warning: Report is 1 commits behind head on main.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| src/aiida/calculations/monitors/base.py | 7.15% | 13 Missing :warning: |
| src/aiida/calculations/importers/arithmetic/add.py | 83.34% | 2 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #7148 +/- ##
===========================================
+ Coverage 29.18% 79.61% +50.44%
===========================================
Files 566 566
Lines 43530 43590 +60
===========================================
+ Hits 12702 34702 +22000
+ Misses 30828 8888 -21940
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
@copilot changes looks good. Please try writing minimal tests
Added minimal tests in commit cac5f4d:
-
test_always_kill_tempfile_cleanup: Verifies the monitor properly cleans up temporary files after execution -
test_parse_remote_data_tempfile_cleanup: Verifies the importer properly cleans up temporary files after parsing
Both tests use monkeypatch to track mkstemp() calls and verify no temporary files remain after execution.