explorer icon indicating copy to clipboard operation
explorer copied to clipboard

Provide complete range of IO operations

Open josevalim opened this issue 2 years ago • 8 comments

Each IO operation must have 8 functions: dump, load, from and to as well as the ! version of them. This is an issue to add the missing APIs. We can break it apart per IO type:

  • [ ] CSV
  • [ ] Parquet
  • [ ] Arrow
  • [ ] NDJSON

Although I recommend tackling #187 before this one!

josevalim avatar Apr 29 '22 09:04 josevalim

We should probably normalize the Rust APIs well.

josevalim avatar Apr 29 '22 10:04 josevalim

What should dump_parquet and dump_arrow return? I guess that dump_csv should return a CSV parsed as a text file, and the others?

kimjoaoun avatar May 05 '22 16:05 kimjoaoun

All of the dump functions should return the contents that we would have found in the file if we used to_format + File.read. So the idea is to skip the writing to disk, and already have the contents in memory.

josevalim avatar May 05 '22 16:05 josevalim

Working on this one. This issue is my current priority.

kimjoaoun avatar May 14 '22 02:05 kimjoaoun

Is there any update on this?

MarceColl avatar Aug 03 '22 14:08 MarceColl

Yes @MarceColl. I'm planning to send the PR that will close this issue later this week.

kimjoaoun avatar Aug 03 '22 14:08 kimjoaoun

Perfect, thanks @kimjoaoun that's awesome :)

MarceColl avatar Aug 03 '22 21:08 MarceColl

Will this read from S3?

isaacsanders avatar Aug 29 '22 11:08 isaacsanders