DIRAC icon indicating copy to clipboard operation
DIRAC copied to clipboard

[9.0] feat: SSHCE export inputs/import outputs as a JSON file to handle more jobs in parallel

Open aldbr opened this issue 2 months ago • 2 comments

In LHCb, we would like to submit a very large number of jobs in parallel via the SSHCE, and we are generally limited to about 300 jobs per cycle because the stamps are generated by the SSHCE and sent through SSH to a batch system. Problem: the line sent is way too long and cannot be executed.

Note: I have not added too many checks for failures for now, I keep it simple and will give it a try in LHCb to see how it works.

BEGINRELEASENOTES *Resources CHANGE: SSHCE export inputs/import outputs as a JSON file instead of using the stdout/stderr ENDRELEASENOTES

aldbr avatar Oct 31 '25 14:10 aldbr

Monitor jobs by batch of 1000 (or more) instead of 100

chaen avatar Nov 10 '25 17:11 chaen

Going through this I discovered a few old problems with the SSHCE, as well as a minor security issue that I will solve in another PR, but these changes are working. I think it's also a good opportunity to finish the scripts I am making for testing the CEs, and finish https://github.com/DIRACGrid/DIRAC/pull/7703

aldbr avatar Nov 13 '25 15:11 aldbr