geoarrow
Issues using Deck with Geoarrow
Hi Kyle, first of all: Amazing work that you are doing! Thank you!
I just found your library and was trying out some of the things. From a blog of you I found earlier, I got this internet speeds dataset: https://observablehq.com/@kylebarron/geoparquet-on-the-web.
And i wanted to make it run with your library but were getting into "not a polygon or multipolygon" errors when loading the parquet via parquet-wasm and then using your GeoarrowSolidpolygon layer. For geoparquet-wasm I run into some errors of the wasm initialization. Maybe some of the parts you are doing here: https://github.com/developmentseed/lonboard/blob/main/src/parquet.ts are missing in the documentation :)
So in the end I tried to implement it with a feather of the centroids directly like you are doing in the blog. I had to adjust it somehow a little as you can see below, but am at least getting the same layer now from what I can see. However, this layer is not showing on the map. If you could have a look, I would really appreciate it. Maybe you can spot the error I am not seeing.
So this is my script:
import React, { use, useEffect, useRef, useState } from "react";
import {Map, useControl} from 'react-map-gl';
import {MapboxOverlay} from '@deck.gl/mapbox';
import {DeckProps} from '@deck.gl/core';
import {ScatterplotLayer} from '@deck.gl/layers';
import 'mapbox-gl/dist/mapbox-gl.css';
import { Table, tableFromIPC } from "apache-arrow";
import { fromArrow, names } from 'arquero';
// import * as d3 from 'd3';
const MAPBOX_ACCESS_TOKEN = process.env.NEXT_PUBLIC_MAPBOX_ACCESS_TOKEN;
function DeckGLOverlay(props: DeckProps) {
const overlay = useControl<MapboxOverlay>(() => new MapboxOverlay(props));
overlay.setProps(props);
return null;
}
async function fetchData() {
try {
// // Fetch Feather file
const resp = await fetch("/data/2019-01-01_performance_mobile_tiles_uncomp.feather");
if (!resp.ok) {
throw new Error(`Failed to fetch Parquet file: ${resp.statusText}`);
}
// convert the response to an Arrow Table
const arrayBuffer = await resp.arrayBuffer();
const jsTable = tableFromIPC(new Uint8Array(arrayBuffer));
let dt = fromArrow(jsTable);
console.log("Arquero table", dt);
const geometryColumn = jsTable.getChild('geometry');
const colorsColumn = jsTable.getChild('colors');
// Loop through each chunk in the geometryColumn.data array
let allCoordinates = [];
geometryColumn.data.forEach(chunk => {
const coordinateValues = chunk.children[0].values;
allCoordinates = allCoordinates.concat(coordinateValues);
});
// Create a new Float64Array with the total length times 2 (x, y)
const totalLength = geometryColumn.length*2;
const concatenatedArray = new Float64Array(totalLength);
// Copy each Float64Array into the concatenatedArray
let offset = 0;
allCoordinates.forEach(array => {
concatenatedArray.set(array, offset);
offset += array.length;
});
// Loop through each chunk in the colors data array
let allColors = [];
colorsColumn.data.forEach(chunk => {
const colorValues = chunk.children[0].values;
allColors = allColors.concat(colorValues);
});
// new Float32Array with the total length times 3 (RGB)
const totalColorLength = jsTable.getChild('colors').length*3;
const concatenatedColorArray = new Float32Array(totalColorLength);
// Copy each Float64Array into the concatenatedColorArray
let colorOffset = 0;
allColors.forEach(array => {
concatenatedColorArray.set(array, colorOffset);
colorOffset += array.length;
});
// Create the data object for the ScatterplotLayer (size: 2 for x, y and size: 3 for RGB)
const data = [{
length: jsTable.numRows,
attributes: {
getPosition: {value: concatenatedArray, size: 2 },
getFillColor: { value: concatenatedColorArray, size: 3 }
},
}];
return data;
}
catch (error) {
console.error("Error fetching data", error);
}
}
function InteractiveMap(): JSX.Element {
const [layers, setLayers] = useState([
new ScatterplotLayer({
id: 'deckgl-circle',
data: [
{position: [0.45, 51.47]}
],
getPosition: d => d.position,
getFillColor: [255, 0, 0, 100],
getRadius: 1000,
beforeId: 'waterway-label' // In interleaved mode render the layer under map labels
})
]);
useEffect(() => {
fetchData().then(data => {
setLayers([
new ScatterplotLayer({
id: 'deckgl-circle2',
data: [
{position: [0.45, 51.47]}
],
getPosition: d => d.position,
getFillColor: [0, 130, 0, 100],
getRadius: 1000,
beforeId: 'waterway-label' // In interleaved mode render the layer under map labels
}),
new ScatterplotLayer({
id: 'internet-speeds',
data: data,
getRadius: 100,
radiusMinPixels: 0.4,
beforeId: 'waterway-label' // In interleaved mode render the layer under map labels
})
]);
});
}, []);
useEffect(() => {
console.log("Layers", layers);
}
, [layers]);
return (
<Map
style={{ height: "100vh" }} // width: "100%",
initialViewState={{
longitude: 0.45,
latitude: 51.47,
zoom: 11
}}
mapStyle="mapbox://styles/mapbox/dark-v9" // {Mapstyles.dark}
mapboxAccessToken={MAPBOX_ACCESS_TOKEN}
>
<DeckGLOverlay layers={layers} interleaved />
</Map>
);
}
const InteractiveMapMemo = React.memo(InteractiveMap);
export default InteractiveMapMemo;
This is the layer it produces:
1: Object { id: "internet-speeds", count: 14, lifecycle: "Initialized", … }
context: Object { mousePosition: {…}, userData: {}, layerManager: {…}, … }
count: 14
id: "internet-speeds"
internalState: Object { needsRedraw: false, needsUpdate: false, usesPickingColorCache: false, … }
lifecycle: "Initialized"
parent: null
props: Object { id: "internet-speeds", getRadius: 100, radiusMinPixels: 0.4, … }
beforeId: "waterway-label"
getRadius: 100
id: "internet-speeds"
radiusMinPixels: 0.4
Symbol(component): Object { id: "internet-speeds", count: 14, lifecycle: "Initialized", … }
Symbol(asyncPropOriginal): Object { }
Symbol(asyncPropResolved): Object { data: (1) […] }
data: Array [ {…} ]
0: Object { length: 3231245, attributes: {…} }
attributes: Object { getPosition: {…}, getFillColor: {…} }
getFillColor: Object { value: Float32Array(9693735), size: 3 }
size: 3
value: Float32Array(9693735) [ 0.6862902045249939, 0.8649733066558838, 0.843987226486206, … ]
<prototype>: Object { … }
getPosition: Object { value: Float64Array(6462490), size: 2 }
size: 2
value: Float64Array(6462490) [ -160.0186157226565, 70.63721610566131, -160.04058837890648, … ]
<prototype>: Object { … }
<prototype>: Object { … }
length: 3231245
<prototype>: Object { … }
length: 1
<prototype>: Array []
<prototype>: Object { … }
<prototype>: Object { data: Getter & Setter, visible: true, pickable: false, … }
state: Object { model: {…}, … }
<prototype>: Object { … }
Thanks a lot!
And i wanted to make it run with your library but were getting into "not a polygon or multipolygon" errors
It's not clear to me; were you trying to render points with the solid polygon layer?
You can only use the SolidPolygonLayer with Polygon or MultiPolygon data, and the geometries need to be in native GeoArrow format, not WKT or WKB.
geoparquet-wasm
geoparquet-wasm isn't really production ready yet. But you can read the README of parquet-wasm, which has most of the same information.
However, this layer is not showing on the map
First try to render with a single color for all points, so then you know the geometry data is at least rendering. You may have to set radius_min_pixels to ensure data are rendered at your current zoom level
Thanks a lot for the quick response! Radius_min_pixels actually helped to notice, that the points are all rendered at origin (see screenshot). I am trying with the centroids created from you python script.
Might this be because I am using the Mapbox Overlay for Deck?
I also tried with the /examples/point setup, but get the following errors:
I don't have a lot of bandwidth to help with this, and screenshots aren't reproducible. You should ensure that you can run the /examples/point demo as is without changing the data source, and then you can inspect what's different with your data compared to its data.
The reason I've focused on Lonboard is that it's easier to create GeoArrow-formatted data in Python than in JS, and there aren't a lot of resources yet for creating this data format in JS proper
Yes, also I think it is more of a problem with deck.gl, as also for random data it renders at origin. I started a discussion there. Thank you anyways! About the demo: I am running it as is in the folder and using the same data with the same script, but it throws those "destructered parameters" errors.
Oh perhaps that's an issue with https://github.com/geoarrow/deck.gl-layers/pull/124. The main library is using deck.gl v9 now but the examples (or at least the yarn.lock) hasn't been updated to use deck.gl v9, so there may be a version mismatch