[FEATURE]: Add MOAB scheduler integration

[shahzebsiddiqui]

Please describe your feature

Let's try adding moab scheduler integration with buildtest. This would require a couple of changes to codebase.

Suggest potential solution

1. We need a top-level executor declaration named moab in buildtest configuration file something like. This will define an executor name my_queue that is mapped to queue name dev.

executors:
  moab:
    my_queue:
       queue: dev

We need to update the json schema in https://github.com/buildtesters/buildtest/blob/devel/buildtest/schemas/settings.schema.json

2. We need to add keyboard for msub in the buildspec to do a simple job submission test such as sleep example

buildspecs:
  hostname_test:
    type: script
    executor: generic.moab.my_queue
    description: run sleep for 5 seconds 
    msub: ["-l nodes=32:ppn=2,pmem=1800mb,walltime=3600"]
    run: |
      sleep 5

This will require updates to the following

• [ ] Update definition.schema.json and update https://github.com/buildtesters/buildtest/blob/devel/buildtest/schemas/script.schema.json.
• [ ] Add implementation for generation of batch scheduler directives see https://github.com/buildtesters/buildtest/blob/8c4e1886dc5daf82f294f3a076757d4990432280/buildtest/builders/base.py#L772-L805
• [ ] Add core implementation for moab executor class https://github.com/buildtesters/buildtest/tree/devel/buildtest/executors and moab scheduler class https://github.com/buildtesters/buildtest/tree/devel/buildtest/scheduler. We can call it moab.py with class MoabExecutor and MoabJob.
• [ ] Also we need to add method for executor checks during configuration loadup see https://github.com/buildtesters/buildtest/blob/8c4e1886dc5daf82f294f3a076757d4990432280/buildtest/config.py#L168-L180. We need a new method to verify moab queues are up and running from ones specified in configuration file. For instance if queue dev is not enabled, we should make the executor invalid

Additional Information

No response

Post question in Slack

• [X] I agree that I posted my question in slack before creating this issue

Is there an existing issue

• [X] I confirm there is no existing issue for this issue