DimArray or DimStack as sink for CSV.read

[tiemvanderdeure]

With https://github.com/rafaqz/DimensionalData.jl/pull/739 it is possible to provide DimStack or DimArray as a sink argument for CSV.read.

However, CSV doesn't pass any arguments to the sink - it just does Tables.CopiedColumns(CSV.File(source; kwargs...)) |> sink. So right now this isn't very useful - it's not possible to pass dimensions.

I think this could be pretty neat to have - maybe we can add a small extension to add this functionality. But I'm not sure what the API should look like (CSV.read is already very extensive). Thoughts?

[rafaqz]

Yes, need to think about the best syntax. And how to get it. What are you thinking?

We could always add our own type like Sink and do CSV.read(table, Sink(DimStack; kw...))

[tiemvanderdeure]

I was thinking that with extension we could dispatch on CSV.read(file, sink::AbstractDimArray; kw...) and then we could split the keywords. Similarly to how we split keywords if you call Raster on a datasource from RasterDataSources. Then we could make something like CSV.read(file, DimArray; dims = (X,Y), name = :col1) work.

[rafaqz]

Yeah, probably thats best from a user perspective. Just slightly annoying to need an extension just for CSV.jl rather than for any table reader

[tiemvanderdeure]

We could also opt for something similar to Rasters - DimArray("mystack.csv", (X,Y)). Which then requires a CSV extension.

So far we don't have I/O in this package, though. But maybe we can include it for simple formats such as csv

[rafaqz]

Yeah, the sink syntax is better as it matches DataFrames.jl

[tiemvanderdeure]

I just realized another option (maybe my favourite) is just to direct users to CSV.File. All keywords of CSV.read are being forwarded to that anyway and it is a table. So

file = CSV.File("dimstack.csv"; limit = 100)
DimArray(file, (X,Y,:category); name = :data2)

just works (found some bugs while testing this but now it's all fixed)

[rafaqz]

Seems easier for us at least :)