kvikio icon indicating copy to clipboard operation
kvikio copied to clipboard

Published 20 hours ago • verified publisher icon rapidsai

S3 support

Open madsbk opened this issue 1 year ago • 5 comments

Implements AWS S3 read support.

Tasks

  • [x] Make the dependency of AWS SDK optional
  • [x] Test with a libcudf io source (https://github.com/rapidsai/cudf/pull/16499)
  • [x] Docs

madsbk avatar Jul 31 '24 15:07 madsbk

Built the C++ wheel locally with https://github.com/rapidsai/ci-imgs/pull/168, and confirmed that it found aws-sdk-cpp, though I did not build the Python wheel or run the tests.

KyleFromNVIDIA avatar Aug 15 '24 20:08 KyleFromNVIDIA

https://github.com/rapidsai/ci-imgs/pull/168 has just been merged. This will be ready to test again once the images are rebuilt.

KyleFromNVIDIA avatar Aug 15 '24 21:08 KyleFromNVIDIA

@wence- thanks for the review, I think I have addressed all of your suggestions

madsbk avatar Aug 23 '24 08:08 madsbk

@KyleFromNVIDIA, what is the status of the building work? Is it ready for review/merge?

madsbk avatar Sep 03 '24 11:09 madsbk

@madsbk Would you please share the status of this work? Are you planning to include AWS C++ SDK changes in 24.12?

GregoryKimball avatar Sep 23 '24 19:09 GregoryKimball

@madsbk Would you please share the status of this work? Are you planning to include AWS C++ SDK changes in 24.12?

The plan is to use libcurl instead of aws-s3-sdk to support S3. First step implemented by https://github.com/rapidsai/kvikio/pull/464.

madsbk avatar Sep 24 '24 06:09 madsbk

Replaced by #479.

vyasr avatar Oct 22 '24 18:10 vyasr