datacomp icon indicating copy to clipboard operation
datacomp copied to clipboard

--output_dir does not do correct thing if --output_dir is a cloud path

Open Vaishaal opened this issue 2 years ago • 1 comments

The datacomp repo is cloudpath aware but open_clip is not, so when we pass a cloudpath like s3:// ... to the open_clip training code it just creates a folder called s3 locally on the master node.

The correct thing to do here is to detect its a cloudpath, give a temporary local directory and enable remote_sync on open_clip

Vaishaal avatar Jul 21 '23 22:07 Vaishaal

  • See https://github.com/mlfoundations/datacomp/issues/56#issuecomment-1734237927

0x2b3bfa0 avatar Sep 25 '23 18:09 0x2b3bfa0