[WIP]Support remote storage (e.g. Amazon s3) in DALI

[xutianming]

Hi all,

I was using DALI with PyTorch recently and am impressed by its excellent performance. Currently all of my training data have to be placed on local SSD storage to use DALI. But in production, we have to read from remote storage, such as Amazon S3. I want to extend DALI to support remote filesystem. Do you think it's possible? Hope to get some early advise from the developers of DALI.

[awolant]

Hi, thanks for the question. If you are willing to contribute we would be happy to help.

For now, we don't see any technical reason for this to be impossible. Although some of our readers are using random access to images (e.x. FileReader) it's not necessary. It's mostly about shuffling patterns after all.

I'm not very familiar with Amazon platform. Let me reach out to some colleagues and get back to you.

[awolant]

@xutianming We discussed this and we do not see any other technical issues, so it's definitely doable feature. We are very interested in adding it, if you are willing to contribute. It might be useful for many other users.

As you mentioned, the way to approach this would be to write something like RemoteFilesystemReader and have Amazon S3 as one of the possible use cases (e.g. GCS being other one). Could you share some more details about your use case and how you plan to implement this.

@jantonguirao also raised a question about mounting Amazon S3 as a storage and using our existing readers. Is that not enough for you for some reason?

[xutianming]

@xutianming We discussed this and we do not see any other technical issues, so it's definitely doable feature. We are very interested in adding it, if you are willing to contribute. It might be useful for many other users.

As you mentioned, the way to approach this would be to write something like RemoteFilesystemReader and have Amazon S3 as one of the possible use cases (e.g. GCS being other one). Could you share some more details about your use case and how you plan to implement this.

@jantonguirao also raised a question about mounting Amazon S3 as a storage and using our existing readers. Is that not enough for you for some reason?

S3-mount is a good choice with almost no extra development. We are maintaining multi-tenant clusters. PyTorch workers are running in dockers and share machines with each other. We found that solutions based on FUSE might get stuck in Linux kernel in this scenario.

Currently, I am planning to extend FileStream to support something like RemoteFileStream . I am working on a demo for proof of concept.

[oleksandrlazariev]

@xutianming were you able to make it work?

[xutianming]

@xutianming were you able to make it work?

Yes, it works and performs well if you have enough bandwidth between your training cluster and the remote storage.

[oleksandrlazariev]

@xutianming how you did it? seems like DALI still not supports it

[xutianming]

@xutianming how you did it? seems like DALI still not supports it

I extended DALI by myself. I plan to make a pull request later.

[oleksandrlazariev]

@xutianming I'm sure it would be great for the whole DALI's community!!!

[goswamig]

@xutianming Did you add the PR ?

[xutianming]

@xutianming Did you add the PR ?

Sorry for the delay. It's still working in progress

[zw0610]

so looking forward to this feature!

[dianaTanasa]

Hi. Is there any news on this feature?

[JanuszL]

Hi @dianaTanasa,

I'm sorry, but there is none from the DALI team's side.

[vivekpayasi]

Is there any ETA on this? Desperately need this.

[JanuszL]

Hi All,

You can try out the external_source operator and see if the data loading performance implemented that way is sufficient (please check especially parallel parameter for the best performance).