Pieter Roggemans
Pieter Roggemans
I added some info about the order of the rows returned + how sorting can be applied to the docstring.
I did some quick tests... and as often is the case, it depends on the drivers involved. I think there are generally 3 types of data sources available through gdal:...
> Because _if_ this is slower, then I am not sure we should provide this feature (if a user wants this, they can always do the `where=f"OGC_FID IN ({','.join(fids.astype(str))})"` themselves)...
> I think providing the functionality - since it allows parity between arrow and non-arrow modes - and documenting (briefly) that performance may vary widely by driver seems like a...
> It's quite similar as what Pieter shows above, performance being very driver dependent, the main summary of those older comments (and from testing shp, gpkg, geojson and fgb): >...
> As a comparison, for Shapefile, it can actually be faster to read the whole file and filter it after reading (and with the arrow reader, this can be done...
Because in theory the sql fid filtering could be fast, I checked on the feasability of speading it up in gdal a few days ago and apparently it wasn't too...
New performance test with GDAL 3.8, that includes the optimization for `fid IN ()` filtering: - .shp and .fgb are now twice as fast with arrow than without - .gpkg...
> Thanks for the updates and your patience here @theroggy . > > Given the performance implications and changes on GDAL side, I think we should limit support for this...
> I guess a warning would be fine, though there is the risk of the user not seeing it and then getting frustrated that things are so slow. But as...