NeMo-Curator
                                
                                
                                
                                    NeMo-Curator copied to clipboard
                            
                            
                            
                        [FEA] Assert that on connecting to cluster you have at least > 1 workers
Is your feature request related to a problem? Please describe.
We need to assert > 1 worker
We need to add a check after this : https://github.com/NVIDIA/NeMo-Curator/blob/9a424c7a498519aeb971a3453ae71447b952b500/nemo_curator/utils/distributed_utils.py#L156