federico-dambrosio

Add SLURM script for launching multi-node Ray clusters with Singularity

2

## Description This PR adds a sample SLURM script and accompanying documentation for running NeMo Curator pipelines on a multi-node Ray cluster using Singularity / Apptainer. Specifically, it: - Adds...

XennaExecutor - RuntimeError: Ray Client is already connected. when RAY_ADDRESS is set to remote cluster

5

**Describe the bug** I'm setting up a Ray cluster and then running a pipeline after setting the `RAY_ADDRESS` variable. **Steps/Code to reproduce bug** `RAY_ADDRESS=ray://$head_node_ip:10001` This is the pipeline: ```python pipeline...