Mark Litwintschik

OOM issue importing an 84 GB OSM file into DuckDB

12

4 of the 5 layers will import without issue and they produced ~84 GB of DuckDB-formatted data. This 5th layer for `lines` causes an OOM. This was run on a...

Specify format to ST_READ when feeding data via stdin?

4

Can `ST_READ` be given a file format or can that only be inferred from a filename? I was hoping something like the following would work but I'm getting ```ERROR 4:...

Detecting a broader amount of invalid geometry

2

I believe you're using GEOS' invalid geometry detector. I've seen a large number of geometry that will pass through its filters without issue but will be rejected by BigQuery. Is...

Can't compile from a folder path with a space in it

1

```bash $ make debug ``` ```bash mkdir -p build/debug && \ cmake -DENABLE_SANITIZER=OFF -DDUCKDB_OOT_EXTENSION_NAMES="spatial" -DDUCKDB_OOT_EXTENSION_SPATIAL_PATH="/Volumes/Seagate/ ./ source/duckdb_spatial/spatial" -DDUCKDB_OOT_EXTENSION_SPATIAL_SHOULD_LINK="TRUE" -DDUCKDB_OOT_EXTENSION_SPATIAL_INCLUDE_PATH="/Volumes/Seagate/ ./ source/duckdb_spatial/spatial/include" -DEXTENSION_STATIC_BUILD=1 -DCMAKE_BUILD_TYPE=Debug -DEXTENSION_STATIC_BUILD=1 -S ./duckdb/ -B build/debug && \...

Unimplemented type for cast (MULTIPOLYGON WKB_BLOB -> POLYGON_2D)

4

Is there a timeline for when casts like the following will be supported? ```sql SELECT ST_GEOMFROMWKB(a.geom) geom2, b.* FROM st_read('geom_xaa.gpkg') a LEFT JOIN us_geom b ON ST_WITHIN(ST_Centroid(ST_GEOMFROMWKB(a.geom))::POINT_2D, b.geom::POLYGON_2D) limit 1;...

Export to GPKG much slower than expected

2

I downloaded boundaries Shapefiles for Spain from https://www.eea.europa.eu/data-and-maps/data/eea-reference-grids-2/gis-files/spain-shapefile ```bash $ ogrinfo -al Spain_shapefile/es_1km.shp | head -n50 ``` ``` INFO: Open of `/Users/mark/Downloads/Spain_shapefile/es_1km.shp' using driver `ESRI Shapefile' successful. Layer name: es_1km...

Performance improvements for Geodatabase imports?

4

I used the latest master branch to convert a 21 GB Geodatabase fileset into Parquet with some light enrichment. It took almost 7 hours exactly on a system with 64...

Avoid scanning entire Geodatabase file

7

It appears an entire scan of a Geodatabase file is needed before a single record can be returned. The following pulls one record from an 870 MB file and takes...

ST_WITHIN doesn't seem to completely filter roads outside of its bounds

8

The following should restrict to streets within the boundary they're within. I exported the table to a GPKG file and brought it into QGIS. ```sql CREATE OR REPLACE TABLE joined1...

Consider turning these into individual point records. Use a flag so this conversion isn't the default behavior. ```bash $ python ~/osm_split/main.py \ sweden-latest.osm.pbf \ --geom-type=other_relations \ --only-h3=820897fffffffff \ --polygon-buildings ```...