Cover job submission API

[gingergeeks]

[laparn]

I would like to insist on this bug. It would be very interesting to have it. Obviously, drmaa (v1) is there and may be used. However, there is one drawback to drmaa : you can only run it as being the user using the program. What would be nice would be to have a launch(user="toto",...) function, where you could choose which user starts the job. It is a key function for any web interface (well there are ways to overcome that, but it will be not very smart).

[gingergeeks]

Once all the new API code is in for 14.11 plus the additional contributions from others, test examples and any major outstanding issues that may affect 14.11 are resolved then this will shortcoming will be addressed. I hope to start coding up a basic submit class over this christmas break.

[ajmazurie]

I second this feature request. After spending a while playing with the API to gather information about running jobs (I couldn't get information about the nodes yet, see https://github.com/gingergeeks/pyslurm/issues/52) I was wondering about how to launch one. I was surprised not to find any reference to it. Is that a planned feature?

Best, Aurélien

[gingergeeks]

Yep, I know it's been open for a long time and I agree but I need to resolve the outstanding issues first.

[giovtorres]

Hi everyone,

I know it has been a while but I wanted to let you know that we are working on wrapping job submit functions and hope to have a decent amount of it wrapped by the time 17.02 is released. For example:

>>> import pyslurm
>>> pyslurm.job.allocate_resources_blocking()
python: job 27205708 queued and waiting for resources
python: job 27205708 has been allocated resources
[debug] job_id:  27205708
[debug] node list:  cn0014
[debug] cancelled job id:  27205708

This is in the dev-16.05-props branch and will be eventually merged into master.

Giovanni

[leezu]

17.02 has been released by now. Is there any plan to go forward with merging the job submit wrapper functions in master? Thanks for your efforts in creating this software!

[giovtorres]

Hi @leezu, I've made some progress with this, but it's been a while. I need to create a branch based on 17.02. I'll try to get to that this week, then you will be able to test out the code. Thanks!

[leezu]

Thanks for the heads up @giovtorres. My university is using version 14.11.11, I'm not sure if your code relies on any APIs released in more recent versions. If not I'll be happy to help testing.

[giovtorres]

That version is a bit older. We've mostly been doing development on 16.05 and 17.02, focusing more on the newer versions of Slurm as they come out. Unfortunately, I don't think we will be backporting any new code into 14.11 versions.

[xuhang57]

@giovtorres Wondering how it goes with the job submission :)

[giovtorres]

Hi, sorry for the delay on this. I did manage to update the development code to 17.02. However, I haven't done much with the job submission portion just yet. I would have preferred to switch over to the newer API before and put efforts there when making newer changes, especially a big change such as job submission, in order to avoid duplication.

I'm currently setting up some automated testing, which should help move things along a bit.

[xuhang57]

Thanks, @giovtorres , just curious, have you got a chance to try to test this PR: https://github.com/PySlurm/pyslurm/pull/65?

Although it is based on slurm 2.6, it should be compatible (or minor modifications needed) with slurm 16.05.5 since no much code change on job submission between these two versions.

[giovtorres]

I did a while back. The C structs have changed some between that version and the current version. I would have to go through it more closely.

[xuhang57]

True and new options added I think. I am trying to make it work with 16.05 but I don't know much about C/Cython. So, don't know how it would go. Thanks for your reply and btw, really like your blog post of wrapping sdiag with Cython. I just wish sbatch would be less complicated 👍

[0xaf1f]

Looks like someone created a project solely to do slurm job submission within Python: https://github.com/brentp/slurmpy

[giovtorres]

Thanks, though it is using subprocess to call sbatch and not through the Slurm API.

[xuhang57]

I pull this #65 PR and make a few modifications for 16.05. It works for some simple scripts. Haven't tested on other options. But generally, that PR should work.

[mstud]

@0xaf1f: if all you need is job submission/control, I would rather use GC3Pie: https://gc3pie.readthedocs.io/en/master/ It doesn't interface the SLURM API though but just wraps the command line tools afaik.

[0xaf1f]

@mstud, thanks. We had a user request installation of slurmpy when we already had pyslurm installed, and I found out from this issue that submission isn't handled by pyslurm. I thought I'd link slurmpy here since people commenting here have wanted this functionality for a few years. But GC3Pie does look better. Thanks again.

[kmanalo]

@giovtorres Just indicating another interested user in using PySlurm for Job Submission

[giovtorres]

Thanks @kmanalo. I hope to focus on this in the coming week.

[alexxxxx]

Hello @giovtorres , Very interesting feature that I can't wait to test :) !!! I can see in the branch "job_submit" pretty recent commits. Is this branch currently in a "testable" state? When do you except to have "beta" release for this feature? Thanks in advance, Alex

[giovtorres]

@alexxxxx very soon, hopefully! I've been chipping away at it. I'm working on processing sbatch directives inside the scripts. Once I have that completed, I'll add some tests and merge it into master.

I think currently the wrap and script s without sbatch directives do work.

[whowell]

Hi @giovtorres. I greatly appreciate this work. I noticed commits as of January 2018. Is there any plan to merge this work soon and/or how usable do you estimate the branch to be at this point?

[pagrubel]

Is this issue still being worked on? I tried out "job_submit" and it ignores #SBATCH directives.

[giovtorres]

@pagrubel Not at the moment by me. Would you like to contribute?

[pagrubel]

If I find a solution, but for now I suppose I'll use subprocess with an sbatch command.