batch-shipyard
batch-shipyard copied to clipboard
Azure
Simplify HPC and Batch workloads on Azure
Bioinformatic tasks may require related pairs (or triplets, etc) of files as input. A typical example is the alignment of paired DNA reads to a reference genome. In this example...
Batch now supports arbitrary ephemeral mountpoints. Remove instances of hardcoded ephemeral mountpoint: - [ ] Mountpoint override/hint in pool.yaml? - [ ] Detect ephemeral disk settings in nodeprep - [x]...
Migrate to the Azure Functions 2.x runtime. - [ ] Create a base Docker image - [ ] Integrate in build and publish to MCR - [ ] Update Azure...
I would like to use the task factorys placeholder variables `{0}` as described [here](https://batch-shipyard.readthedocs.io/en/latest/35-batch-shipyard-task-factory-merge-task/) to set environment variables in my `jobs.yaml` file: ```yaml job_specifications: - id: myID environment_variables: FOO: {0}...
Allow start task (or a named user identity) to drive SSH user provisioning from KeyVault. Should be worked on in-conjunction with #198. Discussion on this feature from #233. - [...
Convenience function wrapping `pool ssh --nodeid/cardinal` to apply to all nodes. - [ ] Add command - [ ] concurrent futures executor to run in parallel - [ ] Migrate...
You can already import shipyard with some small code hacks to use the existing package in scripts directly. However, we should allow for something along the lines of `import batch_shipyard`...
Provide a control on `pool add` and `pool resize` to allocate in phases. This is helpful to workaround certain container registry throttling and throughput limits and where peer-to-peer image loading...
Provide a new option to allow ssh private keys to be stored in KeyVault and referenced in config files. - [ ] `ssh_private_key_keyvault_secret_id` in all `ssh` blocks - [ ]...
Support MSI in credentials - [ ] Credentials config - [ ] Schema update - [ ] Doc updates - [ ] Add limitation note that this cannot be used...