lightning-hpo
lightning-hpo copied to clipboard
Mount s3 drive to notebook
🚀 Feature
Motivation
TODOS:
- [ ] Add Drives Metadata to NotebookConfig
- [ ] Add Drives parsing to the RunNotebookCommand
- [ ] Dynamically create the Drives in the NotebookController and pass them to the Notebook. Propagate the drives down to the works.
API:
lightning run notebook ... --drives {S3_URL}:{MOUNT_DIR} (--env {AWS_CREDENTIALS})
Pitch
Alternatives
Additional context
self.notebooks["1234"].run(
previous_notebook.path,
drives=create_drive(notebook_config.drives)
)