flux-framework
obtain output filename from within batch script
This came up in a user query, and the solution is not exactly intuitive so I'm placing it in an issue so it is searchable and also to discuss if we need some tooling to make this simpler.
Batch jobs have standard output and error redirected to a file of the form flux-{{id}}.out by default. Additionally, any of the other supported MUSTACHE TEMPLATES may be used on the command line to create an output and/or error file whose real path is not known until runtime.
In this case, the user would like to know the output file from within the batch script, but it is not sufficient to just query the jobspec from the parent via
$ flux --parent job info $FLUX_ENCLOSING_ID jobspec | jq .attributes.system.shell.options.output.stdout.path
flux-{{id}}.out
since this only returns the output template, not the final path.
For this case, actual file paths are recorded in the output eventlog in an RFC 24 redirect event:
{"timestamp":1761230474.771312,"name":"redirect","context":{"stream":"stdout","rank":"0","path":"flux-fP5YtyiRcj.out"}}
{"timestamp":1761230474.7713187,"name":"redirect","context":{"stream":"stderr","rank":"0","path":"flux-fP5YtyiRcj.out"}}
To get the pathname from these events is slightly awkward, though. Here's an example shell function that can be used from the batch script to get the current output file (by default stdout):
get_output_file() {
flux --parent job eventlog -f json -p output $FLUX_ENCLOSING_ID \
| jq --arg stream "${1:-stdout}" -r '
select(.name == "redirect" and .context.stream == $stream)
| .context.path
'
}
I'm unsure if this is sufficient or we need some tools here for users to make this simpler.
On tooling, it feels to me like we have a bit of a gap between the functionality offered by flux job info and flux jobs JOBID ....
I wonder if we need a new tool called something like flux-jobattr get|list that fetches arbitrary job attributes? Maybe that could becomes a superset of flux job info and ultimately replace it?
On tooling, it feels to me like we have a bit of a gap between the functionality offered by flux job info and flux jobs JOBID ....
I wonder if we need a new tool called something like flux-jobattr get|list that fetches arbitrary job attributes? Maybe that could becomes a superset of flux job info and ultimately replace it?
My mind began wandering in the same direction, some "aggregator"/"construct useful info" tool similar to flux jobs, but provide things that really should be only for the specific user / job and not published in flux jobs.
I was initially thinking we could just expand flux job info, but a new tool works too.
Would we want to add some things from flux jobs over for convenience for the user? e.g. number of cores, number of tasks, etc.? (maybe recent #7161 id.path). I could see users wanting to use this over flux jobs for information specific to their job?
