explorer
explorer copied to clipboard
Provide complete range of IO operations
Each IO operation must have 8 functions: dump
, load
, from
and to
as well as the !
version of them. This is an issue to add the missing APIs. We can break it apart per IO type:
- [ ] CSV
- [ ] Parquet
- [ ] Arrow
- [ ] NDJSON
Although I recommend tackling #187 before this one!
We should probably normalize the Rust APIs well.
What should dump_parquet
and dump_arrow
return? I guess that dump_csv
should return a CSV parsed as a text file, and the others?
All of the dump functions should return the contents that we would have found in the file if we used to_format + File.read. So the idea is to skip the writing to disk, and already have the contents in memory.
Working on this one. This issue is my current priority.
Is there any update on this?
Yes @MarceColl. I'm planning to send the PR that will close this issue later this week.
Perfect, thanks @kimjoaoun that's awesome :)
Will this read from S3?