aiida-core icon indicating copy to clipboard operation
aiida-core copied to clipboard

Published 20 hours ago • verified publisher icon aiidateam

add file retrieval lists to job template

Open ltalirz opened this issue 3 years ago • 2 comments

In some scenarios, retrieving files after job completion may need to be handled by pushing them back via the compute job rather than by AiiDA trying to fetch them.

By adding the various retrieval/copy lists to the JobTemplate, this is made possible.

ltalirz avatar Sep 12 '22 11:09 ltalirz

@sphuber We needed this information at the scheduler level for an internal plugin. In general, I think this will be necessary for supporting a mode where the compute node is responsible for pushing results back, rather than AiiDA being responsible for pulling results in.

Can you please let me know whether there are any concerns for adding these at the top-level of the job template? If not I can add a test and get this PR ready for review.

ltalirz avatar Sep 12 '22 11:09 ltalirz

In general, I think this will be necessary for supporting a mode where the compute node is responsible for pushing results back, rather than AiiDA being responsible for pulling results in.

How is this information read by the compute node though? The code of the Scheduler plugin and the JobTemplate are executed by the machine that AiiDA is running on.

sphuber avatar Sep 12 '22 12:09 sphuber

Should this maybe be converted into an issue with a feature request describing the use case?

sphuber avatar Oct 27 '22 13:10 sphuber

How is this information read by the compute node though? The code of the Scheduler plugin and the JobTemplate are executed by the machine that AiiDA is running on.

The scheduler plugin uses the information to add corresponding commands to the batch script.

ltalirz avatar Oct 27 '22 14:10 ltalirz

Should this maybe be converted into an issue with a feature request describing the use case?

Yes, I'll do that, but it will require a bit of thought. This PR is a "quick fix" that enables a specific use case.

ltalirz avatar Oct 27 '22 14:10 ltalirz

This PR is a "quick fix" that enables a specific use case.

I see that, but what is the idea of the PR? Were you looking to get it merged? I am asking since you opened it as a draft and like you say it might merit some discussion before changing this. If you agree with opening an issue, maybe this can be closed for the time being

sphuber avatar Oct 27 '22 14:10 sphuber

I will close this for now as the requested feature can be discussed in #5892

sphuber avatar May 17 '23 09:05 sphuber