ray icon indicating copy to clipboard operation
ray copied to clipboard

Published 20 hours ago • verified publisher icon ray-project

[Datasets] Add test coverage for reading from HDFS.

Open clarkzinzow opened this issue 3 years ago • 1 comments

Add test coverage for reading from a pyarrow HadoopFileSystem. Will need to explore mocking HDFS, or starting and managing a local HDFS cluster via a pytest fixture.

clarkzinzow avatar Aug 25 '21 22:08 clarkzinzow

@atomic FYI this PR which is for S3 may serve as an example for HDFS test: https://github.com/ray-project/ray/pull/17158

jianoaix avatar Sep 20 '22 22:09 jianoaix

@jianoaix based on @atomic's bandwidth, we are targeting 11/23 for the completion of this task, let us know if this is reasonable, thanks!

heyitsmui avatar Oct 24 '22 18:10 heyitsmui