clearml-server icon indicating copy to clipboard operation
clearml-server copied to clipboard

The default dataset path

Open jis478 opened this issue 1 year ago • 4 comments

Hi,

Thanks for the awesome project! I'm wondering if I can change the default path for datasets? Currently they are residing in '/opt/clearml/fileserver/{myproject}/.datasets' but it is a root storage so it has been used by ClearML causing all the experiments to halt. I'd like to switch it to one of storage folders which is big enough to hold all of my datasets.

jis478 avatar Aug 02 '22 05:08 jis478

@jis478 To override the default dataset storage location use the output_uri parameter (available in either CLI or SDK)

ainoam avatar Aug 02 '22 10:08 ainoam

@ainoam thanks a lot! By the way, I've uploaded my files to the clearML server using output_uri but another use of the project is not able to download (find) it using Dataset.get API.

It seems like she cannot access the output_uri path in my docker container (because we are using two different containers!)

Do you have the best practice in this situation?

image

jis478 avatar Aug 03 '22 08:08 jis478

Hi @jis478 , what output_uri did you use? is it a mounted dir?

jkhenning avatar Aug 04 '22 08:08 jkhenning

@jis478 did you manage to solve this?

jkhenning avatar Aug 21 '22 20:08 jkhenning