gdalraster icon indicating copy to clipboard operation
gdalraster copied to clipboard

GDAL arrow stream

Open mdsumner opened this issue 1 year ago • 0 comments

I'm pretty keen to get the Arrow stream support in, I tried doing it but got lost in the details of how this package is actually structured at the C++ level.

Dewey Dunnington wrote the required wrapper here, and I'd suggest just returning the arrow stream as the first step:

https://github.com/r-spatial/sf/blob/main/src/gdal_read_stream.cpp

apart from GDAL (3.6.0) and Rcpp that code relies only on "sf/src/gdal_read.h", which has a declaration for an Rcpp list with options passed in by the user in CPL_ogr_layer_setup.

I need some guidance as to where (in files) here to place this code, which I think is easy for you (!), and I'm happy to flesh it out in terms of GDALVector and tests/doc once we have it in. Or, I could branch my naive attempt and ask you to review that?

I have it built into vapour here where I have an option to pull the stream and return a df, or just return the stream object. I got scared off including that because of other problems I was having, and fwiw the code here is a bit out of date compared to Dewey's version:

https://github.com/hypertidy/vapour/blob/main/inst/include/gdalarrowstream/gdalvectorstream.h

https://github.com/hypertidy/vapour/blob/8fb170c6364a0363a9e2793d7b837b216f4e4d79/R/read_stream_internal.R#L10

mdsumner avatar Oct 05 '24 23:10 mdsumner