I want to organize my data in a well-structured hierarchy of folders and give access to specific datasets to specific client

[sriram-dsl]

trafficstars

i have very sensitive data so, i dont want give acces of those datasets to some persons, but they should access other folders

something like in ceph

[ainoam]

@sriram-dsl ClearML does not manage storage access privilege (Well, its enterprise offering offers additional modules to support such capabilities). To address this use case you can configure your storage architecture and privilege policy (e.g. using ceph) and then use clearml-data to log your datasets as references only (see --links) so these privileges are maintained.

Does this make sense?

[sriram-dsl]

Even we are also thinking to do same If possible can you share, any related documentation

On Wed, 6 Dec, 2023, 00:02 Noam Wasersprung, @.***> wrote:

@sriram-dsl https://github.com/sriram-dsl ClearML does not manage storage access privilege (Well, its enterprise offering offers additional modules to support such capabilities). To address this use case you can configure your storage architecture and privilege policy (e.g. using ceph) and then use clearml-data to log your datasets as references only (see --links https://clear.ml/docs/latest/docs/clearml_data/clearml_data_cli#add) so these privileges are maintained.

Does this make sense?

— Reply to this email directly, view it on GitHub https://github.com/allegroai/clearml/issues/1165#issuecomment-1841380408, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVOFMKN6TB5GJWE6KRP2WDYH5SDBAVCNFSM6AAAAABABCSI4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRGM4DANBQHA . You are receiving this because you were mentioned.Message ID: @.***>

[ainoam]

@sriram-dsl This is the relevant documentation: --links parameter to clearml-data add. What are you missing?

[sriram-dsl]

I am following all those CLI commands, but I want some datasets to be publicly available, some datasets accessible to certain clients, some internally within our team, and some privately accessed by me.

For this use case, we can use access control in ClearML, but it currently limits access control to the GUI only. Once someone has GUI access, they can view dataset IDs and even generate credentials. Therefore, I want to restrict their GUI access, particularly for certain datasets.

On Thu, 7 Dec, 2023, 23:00 Noam Wasersprung, @.***> wrote:

@sriram-dsl https://github.com/sriram-dsl This is the relevant documentation: --links https://clear.ml/docs/latest/docs/clearml_data/clearml_data_cli#add parameter to clearml-data add. What are you missing?

— Reply to this email directly, view it on GitHub https://github.com/allegroai/clearml/issues/1165#issuecomment-1845786616, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVOFMM6QEFCDSSJ3J7CZEDYIH4L7AVCNFSM6AAAAABABCSI4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVG44DMNRRGY . You are receiving this because you were mentioned.Message ID: @.***>

[ainoam]

@sriram-dsl The point I'm trying to make, is that using --links will make it so the data itself is not stored in ClearML but rather on your storage of choice, and so regardless of having access to the ClearML dataset your storage access policy will dictate who can actually access the data (i.e. I can see the dataset, but I can't access the files unless I have privilege to those files on the storage server). Role based access control is available as part of the ClearML enterprise offering.

[sriram-dsl]

I agree with you. I have used it for S3, and it worked. Our team is looking to launch a Ceph cluster and push the data to Ceph to limit access. Currently, we are using NAS for our storage. I want to know how to launch Ceph on NAS and mount the Ceph-deployed NAS folder on opt/clearml/data/fileserver.

On Sun, 10 Dec, 2023, 22:16 Noam Wasersprung, @.***> wrote:

@sriram-dsl https://github.com/sriram-dsl The point I'm trying to make, is that using --links will make it so the data itself is not stored in ClearML but rather on your storage of choice, and so regardless of having access to the ClearML dataset your storage access policy will dictate who can actually access the data (i.e. I can see the dataset, but I can't access the files unless I have privilege to those files on the storage server). Role based access control is available as part of the ClearML enterprise offering.

— Reply to this email directly, view it on GitHub https://github.com/allegroai/clearml/issues/1165#issuecomment-1849017009, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAVOFMMVZMBJWY5KDDFOZCLYIXRNZAVCNFSM6AAAAABABCSI4WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBZGAYTOMBQHE . You are receiving this because you were mentioned.Message ID: @.***>

[sriram-dsl]

If I receive extended help, like relevant documents, it is appreciated. If not, no issues either. We are currently occupied with other matters, but we will address it soon.