[WIP] cmdline/submitor

[Roy-Kid]

Hi @skrawcz and other developers

Here, I start a new pull request to introduce new features for cmd line execution and submit jobs to HPC queue management. Since both features require external execution, we can implement them together.

First, we implement @cmdline decorator and related executor, which can execute locally(serial) and remote(on parallel).

@cmdline
def test_external_job(input:Any)->Any:
    # before execute
    output = yield "cmdline"
    # process output
    result = post_process(output)
    return result

Second and similarly, we implement @submit to submit a job to HPC queue system

@submit(name="local_slurm", type="slurm")  # initialize a new submitor which represents a HPC
def test_slurm(input:Any)->Any:
    # before execute
    output = yield {  # slurm config and cmd
        "job_name": "test_slurm",
        "n_cores": 4,
        "memory": 1000,
        "max_runtime": 100,
        "work_dir": ".",
        "monitor": True,
        "cmd": f"echo {message}"
    }
    # post process
    return analysis_result

Third, we can block within a node and wait for an external job to finish by using Monitor, which internally pools status of jobs at a certain interval.

Using two decorators, we can hold hamilton DAG but execute jobs outside.

Changes

How I tested this

Notes

Checklist

• [ ] PR has an informative and human-readable title (this will be pulled into the release notes)
• [ ] Changes are limited to a single goal (no scope creep)
• [ ] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
• [ ] Any change in functionality is tested
• [ ] New functions are documented (with a description, list of inputs, and expected output)
• [ ] Placeholder code is flagged / future TODOs are captured in comments
• [ ] Project documentation has been updated if adding/changing functionality.