Streaming via ssh across clusters

[gaow0007]

🚀 Feature Request

I want to use streaming to access a remote datacenter via ssh with certain privacy-related permission.

Motivation

I want to use a separate cluster to store the dataset, and want to set the permissions for different images in the datasets for different users.

[Optional] Implementation

Cloud you provide some potential API or examples which might be helpful to this feature?

Additional context

[karan6181]

Hi @gaow0007 , could you please explain your use-case a bit more in detail ? Currently, Streaming dataset support AWS S3, GCS blob Storage, and OCI blob storage which downloads the dataset on the fly onto a remote instance. Streaming Dataset automatically authenticates based on your cloud provider and provided authentication and perform either download/upload the shard files.

[gaow0007]

Sure.Considering that dataset is split into different partitions, and some users can access the entire datasets, however others only can access partial partitions due to privacy issues.

Additionally, is it possible to provide a ssh example to use streaming dataset to facilitate local-mode debug

[snarayan21]

Hey @gaow0007, support for sample-level permissioning for datasets is not currently planned on being supported by Streaming. Closing out this issue since it's been inactive for a while.