Investigate usage of job sets at agent job submission level for matchmaking of specific workflows with heterogeneous resources

[khurtado]

Impact of the new feature Job sets allows creating a group of jobs (with different cluster Ids) for a specific set name. https://htcondor.readthedocs.io/en/latest/users-manual/job-sets.html

This issue is incomplete, but this feature was mentioned at some point during the USCMS AHM and it could be used for different things, so I'm just going to go ahead and describe the feature here. We can think about how we would use it later (e.g.: matchmaking of specific workflows with heterogeneous resources?).

Is your feature request related to a problem? Please describe. When we submit jobs, we pull all available work from different workflows and submit them in 1 job cluster (this is also more efficient from the submission point of view). However, it's not easy to know what jobs belong to what workflow, unless you look directly into the job classads. Using the condor native job sets feature, would allow us to list all the condor jobs in a workflow and potentially edit, change priority of all jobs at once, etc.

For example, to list the status of all jobs within a set of jobs/workflow, you do:

htcondor jobset status MyJobSet

MyJobSet currently has 3 jobs idle, 5 jobs running, and 0 jobs completed.
MyJobSet contains:
    Job cluster 1234 with 4 total jobs
    Job cluster 1235 with 4 total jobs

Describe the solution you'd like Implement job sets in the condor plugin.

[amaltaro]

@khurtado Kenyi, I see this ticket is marked as "In progress" but currently unassigned. Should you self-assign it or should it be moved back to ToDo status?

[khurtado]

Let me move it back to Todo