tracking job spec for jobs submitted with flux mini commands

[ryanday36]

Most (all?) resource management software allows users to include the parameters for their job allocations in their job scripts using a pragma syntax (e.g. #SBATCH, #BSUB, etc). This effectively allows users to record everything that they used to submit the job in one place, which can be very useful for documenting older work, re-submitting the same work, sharing with others, etc.

The model in Flux is that the job parameters are kept in the jobspec separately from the job script, but users of the flux mini commands never work directly with their Jobspec, only the flags that they include on their command line. It would be useful for users of flux mini commands to be able to track their command line flags in a file for future reference.

[ryanday36]

Some possibilities for dealing with this:

1. have users convert job scripts to heredocs. This is what I suggest in my LC tutorial (https://hpc-tutorials.llnl.gov/flux/section3/). It's pretty clunky.
2. Do the pragma thing: update flux-mini.py to read #FLUX lines (or similar) in a users job script and parse those in addition to any command line arguments.
3. allow users to specify a separate file with their command line arguments (e.g. using argparse's fromfile syntax).

[garlick]

I tend to like 3, if you think that would fly. Coupling the run parameters to the script with pragmas encourages making a copy of the script each time parameters are tweaked. Unless you quickly develop some way to put the reusable portion of the script under version control, that can lead to a mess. (Feel free to correct me if that impression is incorrect and real users don't behave like...me)

I wonder if instead of putting flux mini arguments in a file though, could we use YAML jobspec? We already turn the arguments into jobspec anyway - that is mostly what the flux mini commands do. (try flux mini --dry-run and pretend the JSON output is YAML :-) This would be more in keeping with the original Flux design.

Note that we do preserve the submitted jobspec for each job. Maybe for batch jobs we could have that pop out to a file like the standard output? Maybe some tooling (perhaps within flux mini) could make the jobspec a little easier for humans to work with? For example, converting a finished jobspec back to a fragment that could be hand edited?

A related topic that we have to work on anyway is chaining jobspec so that we can allow a user to tweak jobspec parameters of a job once it's already in the queue. Maybe that capability could turn into some kind of template tooling that would be useful for manipulating jobspec before submission?

[ryanday36]

Since I mostly interact with the people who are less experienced and need more help, my sample is probably biased, but I suspect you may be overestimating the usage of version control in our general user population :) I think this boils down to two use cases:

1. user ran something sometime in the past, and wants to check on exactly what resources they submitted on. Having the flux mini command write something out to stdout or somewhere in the same directory as the run script would cover this.
2. user wants to re-run something exactly as they ran it before or run a job that maybe has some changes in the job script, but will request the same resources as something they ran before. This would also be covered by the flux mini command writing out a Jobspec, as long as that Jobspec can be used as input for another flux mini command.

So, I think that what I would imagine for the 'separate file' option would basically be to add something like a '--jobspec=FILE' flag to flux mini that would allow users to put all of their flags in a file or put the equivalent amount of YAML in a file and re-use that across multiple flux mini commands. The thing that I don't want would be for the user to lose the mini-ness of flux mini if they use a file rather than a set of command line flags. So, if they're able to get what they want with something like:

flux mini batch -N2 -n4 myscript

they should alternatively be able to have an argument file that only defines the number of nodes and the number of tasks.

[garlick]

We can almost do that now with

$ flux mini batch -N2 -n4 --dry-run myscript >myscript.json
$ flux job submit myscript.json

myscript.json is the full jobspec in JSON form, not very human friendly. However since we have YAML support built into our python jobspec class that is used by flux mini, we could easily fix that.

(Note: if you find yourself wanting to examine that json file, a handy tool is jq <myscript.json)

[garlick]

Although a user can access R and jobspec now using flux job info, it seems sort of appealing to me to have the rank 0 job shell place a copy of R and jobspec in the same directory as the stdio files for batch jobs. It could be a shell option that flux mini batch sets and then maybe implemented by a shell plugin?

Then if flux goes :boom: the user has something to hold on to :-)

[grondo]

have the rank 0 job shell place a copy of R and jobspec in the same directory as the stdio files for batch jobs.

Another idea would be to dump jobspec and R into the output file, so that these files are all kept together.

Perhaps we could define a "batch job output file" format in JSON which could capture jobspec R and output eventlogs in one file, with a tool that could be used to grab individual components from the file? Since the batch script is captured in jobspec, a flux mini command could be used to submit identical jobs from a previous batch job file by starting from the embedded jobspec.

[ryanday36]

I like the idea of being able to use a flux mini command to submit an identical job from a previous batch job. If a user used flux mini to submit the original job, it seems like they should be able to use flux mini again to re-submit.

I'm not so sure about the idea of combing all of the jobspec and such in one JSON file with the output. I don't really like the idea of users having to learn/use another flux command to view their output. That said, LSF does print the job script in its output files and it's not too distracting, so it probably could work as long as its formatted in a way that's relatively human readable.

The more I think about this discussion, the more I think that I've misstated my actual concern. I think that the arguments about keeping a record of what was run is mostly my justification for what I'd actually like to see, which is a way for users to be able to put their flux mini arguments in a file rather than specifying them on the command line. It's what they're used to with #SBATCH/#BSUB/etc pragmas and it has advantages over a full jobspec in terms of being human readable and, especially, writable. I'm certainly not married to a pragma approach, but I think that having a mechanism by which a user can write their job parameters in a file, then use that file to submit one or more jobs would be better than one where the user submits their first job with a bunch of flux mini command line flags, then has to re-submit that job via some other mechanism or find their way through a full jobspec if they want to change one parameter before re-submitting.

[garlick]

OK then, my vote would be for your original argparse fromfile suggestion over script pragmas, as that avoids the coupling concern I mentioned earlier.

That wouldn't necessarily preclude dumping out a machine readable object at the end, as suggested by @grondo. Such a thing could be really useful for post-mortem debugging and perhaps for formal provenance of results.

[chu11]

Since time has passed since this discussion thought I'd throw out a random idea that came to me:

1. With #4336 looming, we would have a permanent way for users to lookup prior jobspecs, even those dumped from memory

2. So perhaps a flux mini submit --run-like-job=<jobid> could be done? i.e. whatever args used in this prior job, use it again.

2A) And anything specified on command line could overwrite?

[garlick]

While those pragmas have always seemed "wrong", the argument against them feels academic at this point. Given all the other actual problems we have to solve and the relative simplicity of resolving this, I vote we go ahead and add support to flux mini batch if it makes people happy. A simple implementation would just translate #FLUX prefixed lines into command line arguments, with the actual command line overriding.

Do we need a separate issue to track using an existing job as a template for submitting a new one? Seems like we got slightly off track in the discussion.

Edit: speaking with @grondo offline, a --ignore-pragmas (or similar) option to flux mini batch might be useful to have. Also, maybe an option to change the pragma prefix, in case the batch script is in a language that doesn't treat # as a comment?

[garlick]

I was just toying with this idea this morning and wondered if it would be a slight improvement over scraping comments.

Let the batch script accept an option like --pragma-dump that tells it to print a list of flux mini batch options and exit, then add a flux-mini-batch --enable-pragma option that tells it to run the script with that option to obtain its command line options. Example:

#!/bin/bash

source ./flux-pragma.sh

flux_pragma_begin

flux_pragma -N1
flux_pragma -overbose=2
flux_pragma --queue batch
flux_pragma --time-limit=1m
#flux_pragma --time-limit=5m
flux_pragma --job-name=myjob

flux_pragma_end
# The actual job begins here
sleep 30

The bash helper script flux-pragma.sh is optional and just contains:

shopt -s expand_aliases

flux_pragma_append() {
	flux_pragma_args+=($@)
}

flux_pragma_dump() {
	echo ${flux_pragma_args[@]}
}

if test "$1" = "--pragma-dump"; then
	alias flux_pragma_begin='declare -a flux_pragma_args=()'
	alias flux_pragma='flux_pragma_append'
	alias flux_pragma_end='flux_pragma_dump; exit 0;'
else
	alias flux_pragma_begin=':'
	alias flux_pragma=':'
	alias flux_pragma_end='shopt -u expand_aliases'
fi

Some advantages:

• There are no flux-specific comment parsing rules to worry about
• A pragma can be commented out with confidence that the comment will have the desired effect
• Option values can be computed by the shell script if desired.

Not sure if if this would be helpful or acceptable but wanted to throw it out there.

[ryanday36]

I think that something like this would work if the flux_pragma_begin, flux_pragma_end, and flux_pragma aliases were recognized and parsed by the flux mini batch command directly rather than requiring users to track and source the flux-pragma.sh helper script in all of their job scripts in order to use those aliases. I can see the value of having the user delimit where the pragmas are in their script to avoid confusion with comments and parsing, but I think that the flux-pragma.sh would just be an extra step for the users that would lead to confusion and errors.

[garlick]

This approach requires that the user specify the --enable-pragma option to flux mini batch if they have pragmas, and then to use them correctly. So a bigger problems may be

• If the option is skipped, pragmas areignored
• If the option is provided and pragmas are missing (or if just the flux_pragma_end directive is missing) then the script will blow up/hang trying to submit stuff outside of the user's allocation.

The upside is it just asks the script to do something simple to provide its parametric info, which scripts are good at, as opposed to this scraping which has flux digging around in bash's underwear drawer or whatever. It feels wrong!

Back to the drawing board.

[abbotts]

To reframe and concretize this a little bit: the real benefit of a batch directive analog is to support legacy workflows.

There's lots of existing workflows (for example the E3SM CORAL-2 benchmark) that have infrastructure setup to write and submit a traditional batch script with an unordered set of directives.

Sensible or not, the infrastructure differentiates between properties that can be "batch" parameters vs "job" parameters, as a legacy of systems where the batch system and the job launcher were entirely distinct entities. Working around that distinction would take a significant reworking of the case setup and submission system. E3SM isn't unique in this regard, although it might be one of the more egregious.

Even the flux_pragma_begin/flux_pragma_end solution or a heredoc would require some re-plumbing and a unique flux template for the case run scripts, since the existing template doesn't expect to bookend the batch directives in any way.

[garlick]

Thanks @abbotts !

This issue is now a high priority one on our project board so it has a reasonable chance of being addressed soon.

Sorry the discussion was all over the place above. We've internally resisted implementing these pragmas for a long time but it appears their time has come. Details TBD.

[grondo]

Note: RFC 36 has been merged and describes the specification for solving this issue. There is still time to adopt feedback into the initial RFC before the implementation is posted in a PR (probably by the end of this week)