flux-core
flux-core copied to clipboard
flux-framework
job-list: handle jobspec-update event
Problem: Missing jobspec attributes may be filled in by the jobspec-update event, but job-list does not parse that event.
Edit: reference pr #4386
Related: the shell fetches the jobspec directly but not the eventlog, yet jobspec-update events could be posted for things the shell should parse, like shell options. Should job-info take care of this automatically when the jobspec is fetched? E.g. return jobspec with the modifications applied?
Oh, good thinking!
I had in the back of my mind that jobspec-update events would be for the instance only and would be ignored by the shell, though the default shell options use case is appealing. If we could do signed jobspec-update events, then this could be used by users to modify jobspec after submission, and in that case it would be nice if job-info could apply the updates so that a request for the jobspec would return the current version.
This reminds me, though, that we still need to address #2814
Edit: It may be obvious, but we should be clear that the reason the job shell might ignore jobspec-update events is that they invalidate the signed J submitted by the user. In an untrusted instance, we don't want the instance forcing the job shell to run something the user didn't authorize, so we shouldn't bake that into the design. Granted, at this point the job shell does fully trust the instance since it directly fetches jobspec and not J.
Over in #4494 we came to the conclusion that, at least for the current need of assigning jobspec defaults, we would modify the jobspec before it lands in the KVS or the job-manager. The original, signed jobspec would be retained in J unmodified.
In PR #4523, the job shell was modified to unwrap the original jobspec from J.
Then in #4520 it is proposed that we drop the environment from the jobspec stored in the KVS.
PR #4529 relocates the jobspec-default plugin to job-ingest.
So in summary, our design for jobspec defaults drifted a bit since this issue was opened and it is no longer urgent for that use case that we support the jobspec-update event in the tools. However, we should leave this issue open as we still may need jobspec-update for other user cases like #2799.
Pinging this issue since we're looking into modifying a limited set of jobspec attributes post-submission. So I think the priority of this issue should be bumped up again.
Hadn't thought about this issue in awhile, mostly b/c of the short/medium term need. But if need is super apparent given recent PRs, can obviously bump up.
