Add vectorgrid support

[emilhe]

Add wrapper(s) of leaflet vectorgrid,

https://github.com/mhasbie/react-leaflet-vectorgrid

[prl900]

Hi @emilhe, I wonder how much effort or difficult would be to add this feature to the library. Being able to display vector tiles would be quite useful for working with dense polygon data in Dash applications. Thanks!

[emilhe]

I don't think it will take a crazy amount of time. At least not to implement the basic functionality (:

[henrygsys]

Hi, is this something that is the pipeline to do soon? I would like to be add Mapbox Vector Tile sources to Dash Leaflet but it seems they are not supported. Are there any workarounds for this?

[emilhe]

@henrygsys It's one of the top 3 (larger) features to be requested. I haven't got an exat timeline, but it would help if you could post details here with respect to your usecase (e.g. the URLs that should be loaded), so that I can use it for testing.

[henrygsys]

Sure.

Essentially, I am using Mapbox as a tileserver to store some layers of vector data, mostly global sets of polygons to show protected areas and that sort of thing. The urls follow the below convention and they are added to mapbox using the vector tiles url.

f"https://api.mapbox.com/v4/{tile_set_name}/{{z}}/{{x}}/{{y}}.mvt" https://docs.mapbox.com/api/maps/vector-tiles/

It is possible to add these using the general Dash map figures by passing it as a mapbox layer source using the below. But as i understand it, these do not render when using dl.TileLayer.

fig.update_layout( mapbox={ 'accesstoken': MAPBOX_TOKEN, 'style': style, 'zoom': 2, 'layers': map_layer }

[emilhe]

I did a bit of research on possible options yesterday. It seems that VectorGrid doesn't support mapbox tiles (and it also seems abandoned). I then stumled upon maplibre-gl-leaflet, which supports mapbox files, but doesn't have the same amount of configuration options as VectorGrid. If anyone has experience with mapbox/vector tiles using "raw" leaflet, I would be gratefull for some input on which plugin to choose here in 20203 :)

[ivvnnnn]

@emilhe It seems Vector Grid can support Map Box Vector tiles, if call VectorGrid.Protobuf (link: https://leaflet.github.io/Leaflet.VectorGrid/vectorgrid-api-docs.html). In Leaflet it works. React-leaflet support protobuf too (link: https://github.com/mhasbie/react-leaflet-vectorgrid). May be, it help

[emacollins]

Hi, just wanted to follow up on this thread and the proposed vectorgrid support. Wondering if there are any updates (or potential workarounds?). I am trying to display a very large GeoJSON and it is really slowing down/crashing the app.

Seems the alternative in this situation would be to use a vector tile server (like Tegola).

[emilhe]

While the discussion was active (~ a year ago), i played a bit around with vectorgrid and a few of the alternatives (they differ in their interfaces, which formats are supported, where/how/if slicing is supported, etc.). However, it was never settled which component(s) to add to dash-leaflet.

To my knowledge, there hasn't been any work since then. I haven't encountered any use cases in a work related context, and I thus haven't been able to prioritize development time on the topic.

[Eric-UW-CRL]

First off, really love this project and what it enables as far as building mapping applications with Dash!

It seems like adding the VectorTileLayer would be a huge unlock to the functionality and bring together the ability to display raster and vector data.

I see there is a branch started for this. Where did you leave off or run into issues with this? I would love to help contribute but am still familiarizing myself with the codebase and don't have experience with JS.

[emacollins]

Hey @emilhe , just wanted to follow up. I am curious what issues (if any) you ran into on the vector grid branch? Would love to pick up where you left off and try to get this feature added. I am a beginner in TS and JS.

[emilhe]

The different components address slightly different use cases (various formats, slicing of geojson, etc.) and hold different features. I didn't face any technical issues, but rather the issue of choosing which component to adopt. Do you have a favorite?

[emacollins]

I think that the ideal component to adopt would be the Leaflet VectorGrid, mentioned above by @ivvnnnn using VectorGrid.Protobuf.

In practice it should function quite similarly to TileLayer, except that it will be expecting .pbf responses based on the [Mapbox Vector Tile Spec](MapBox Vector Tile Specification).

It seems like this vector tile spec is widely used, and plays well with PostGIS databases, which has built-in functionality to generate these tiles from geometry data. There are open source tile servers like Tegola that anyone could use to serve their own data directly to dash-leaflet. This way users can now serve vector and raster data to dash-leaflet, which would really round out and enhance the project. In my own project, the biggest bottleneck has been trying to load huge GeoJSONs containing my vector data.

I have found dash-leaflet a joy to use and is a really great project, and well documented! Thanks for all the work you have put into it. @emilhe

[emilhe]

@emacollins it was also the first library (or rather, it's react wrapper) that I looked at. However, with the latest commit dating back years, it seems unmaintained, which makes me hesitant to port it to dash-leaflet. Another project, which more recent development activities is VectorTileLayer , though it lacks some features, and I haven't found any React bindings. There's also react-leaflet-vector-tile-layer , which again has a slighly different feature set, and probably more. Do you have an overview of, which of these options would support your usecase(s)?

Thanks! I am happy that you like it 👍

[emacollins]

After looking through both of them, it looks like VectorTileLayer is more comprehensive and has the functionality that I think fits this use case. The other one claims to support vector tiles, but the documentation is a bit sparse and it doesn't seem like it implements what I am thinking of. Maybe it would work but there is not much there to know.

I realize now that VectorGrid was not part of React-Leaflet. I assume you are referring to this react wrapper that seems unmaintained .

[emilhe]

Reviewing the components in their current state, this is also the option that seems most promising to me at a glance. When I can find the time, I'll start there. If you manage to get something working in the mean time, please create a PR and link it here.

[emilhe]

I have implemented the bare minimal bindings to get data flowing into the VectorTileLayer library from Dash. There is still a lot of work before the component is ready for publication, but it's a start. Here is a sample app,

import dash_leaflet as dl
from dash import Dash

app = Dash()
app.layout = dl.Map(
    [
        dl.TileLayer(),
        dl.VectorTileLayer(url="https://openinframap.org/tiles/{z}/{x}/{y}.pbf", maxDetailZoom=6, style={}),
    ],
    center=[56, 10],
    zoom=8,
    style={"height": "50vh"},
)

if __name__ == "__main__":
    app.run_server(debug=True)

that yields,

where the blue paths are rendered from vector tile data (default styling, default rendering). If you run

pip install dash-leaflet==1.0.17rc1

you should be able to run the example. I have crated a draft PR with the WIP code, https://github.com/emilhe/dash-leaflet/pull/255. @emacollins could you try it out? Testing the (initial) implementation toward a real use case would help map out were to go next in the implementation.

I don't have any (work-related) use cases at the moment. But I was thinking about wrapping up a complete example in docker compose for visualizing huge GeoJSON files, possibly using Tegola as backend. But that'll have to wait for anoter day :)

[emacollins]

This is amazing, thank you! @emilhe. I will put some time in at some point this week to test it out as well. I can put together a simple app using Tegola and some data from my use case using a PostGIS database. Can help out a building out a complete example.

[emilhe]

I have just pushed a new update which includes bindings of event handlers,

pip install dash-leaflet==1.0.17rc2

Building on the previous example, the syntax would be,

import dash_leaflet as dl
from dash import Dash
from dash_extensions.javascript import assign

eventHandlers = dict(click=assign("function(e, ctx){console.log(e); console.log(ctx);}"))

app = Dash()
app.layout = dl.Map(
    [
        dl.TileLayer(),
        dl.VectorTileLayer(url="https://openinframap.org/tiles/{z}/{x}/{y}.pbf", maxDetailZoom=6, style={}, eventHandlers=eventHandlers),
    ],
    center=[56, 10],
    zoom=8,
    style={"height": "50vh"},
)

if __name__ == "__main__":
    app.run_server(debug=True)

I guess next step is to decide, if default event handler bindings should be included (similar to the GeoJSON component).

[zifanw9]

I am experimenting with vector tile. To me, an event handler is useful for retrieving info of the clicked feature (in my case, I want to display popup), and the current event handler is sufficient with the following steps:

Step 1

buildingEventHandlers = dict(click=assign("""function(e, ctx) {
                                                ctx.setProps({
                                                    n_clicks: ctx.n_clicks == undefined ? 1 : ctx.n_clicks + 1,  // increment counter
                                                    clickData: {
                                                        latlng: e.latlng,
                                                        properties:  e.layer.properties
                                                    }  // collect data (must be JSON serializeable)
                                                });  // send data back to Dash
                                            }"""))

Step 2: Initialize the tile layer and a popup inside a map container (to hide it when the page is initially loaded and the user has not clicked on any feature yet, I also set its z-index to very low)

dl.Pane(style=dict(zIndex=1),name="bottompane",id="paneforpopup"),

dl.Popup(position=[34.432546, -119.699944],children=Purify(id="buildingpopupcontent"),id="buildingattributepopup",pane="bottompane"),

dl.VectorTileLayer(url="<TiPG Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}",style=building_style,filter=building_filter,eventHandlers=buildingEventHandlers,id="building"),

Step 3: Define the callback function to update the popup when user clicks on the vector tile layer

clientside_callback(
    """
    function(nclicks,click_data) {
        if (document.querySelector(".leaflet-popup").parentNode.style.zIndex != 1001) {
            document.querySelector(".leaflet-popup").parentNode.style.zIndex = 1001;
        }
        const popupcontent = `Res: ${click_data.properties.res}<br>` +
                             `ComInd: ${click_data.properties.comind}` ;
        return [[click_data.latlng.lat,click_data.latlng.lng],popupcontent];
    }
    """,
    [Output("buildingattributepopup","position"), Output("buildingpopupcontent","html")],
    Input("building","n_clicks"),
    State("building","clickData"),
    prevent_initial_call=True
)

Not sure if my example here will help on deciding whether to include a default event handler.

Edited Also can I suggest to introduce a hideout property to the vector tile? Not sure if it is convenient to include hideout, but I think this property is useful so that it will allow users to filter features interactively.

Another concern about vector tile/grid is memory usage on clientside browser. I intend to use it as a way to render large dataset (which GeoJson method consumes a lot of memory). When I use Martin tile server docker deployment on Azure App Service, the dash app initially opens with 736MB memory on Chrome, and at some point reached 1.3 GB and later lower to 232 MB memory. Later I try TiPG tile server deployed on render.com $7 plan, I see the memory usage is about 471-520 MB for my dash app. A benchmark comparison with a different map service, using ArcGIS ExperienceBuilder (probably Esri's own tile server) to render the same data consumes 575-711 MB memory. Not sure if anyone has insights on tile server choices or perhaps good practice on reducing memory usage of dash app.

Edit 2 Another note: when I try to map buildings:

1. If I do not specify minDetailZoom=13, tiles corresponding to areas with many small buildings fail to show properly if zoom level is below 13
2. If I do specify minDetailZoom=13, when I zoom to zoom level 11, the memory usage can reach around1 GB even with TiPG tile server. I know this is more of a tile server issue rather than dash issue (Esri Experience Builder in contrast seems to apply a good generalization when zoom to low zoom level), but I feel it is still necessary to mention the issue here. My current mitigation strategy is to use the following filter so that the layer will not display at low zoom level

building_filter = assign(
    """function(properties,layername,zoom) {
        if (zoom >= 11) { return true; }
        else { return false; }
    }
    """
)

[zifanw9]

Hi @emilhe, in PR#255 line 100 of src/ts/react-leaflet/VectorTileLayer.ts, vector tile layer does not seem to have method setUrl(url), and it causes error if I try to update the url of the layer. Not sure if there can be any fix to this issue.

I also just realize it might be possible to do filtering on vector tile layer without introducing hideout property, but this requires updating the url of the layer. This will involve using CQL if the tile server (e.g., TiPG) supports it, a few examples:

url="<Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?filter-lang=cql2-text&filter=res IS NULL AND comind IS NULL"
url="<Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?filter-lang=cql2-text&filter=res IN ('0','1')"

I feel this vector tile layer is ready for release if its url update method can be fixed. Hideout property might still be useful if we want to change the style, but this can just be future enhancement maybe. Thank you!

[zifanw9]

I created a PR in VectorTileLayer repo to add setUrl method. Not sure whether and when my PR is ok to merge though.

In terms of dash-leaflet side, the following changes might need to be considered:

• Line 96 of VectorTileLayer.ts updateGridLayer(layer, props, prevProps) probably needs to be removed. Sometimes with this line included I see exception and malfunction (but I am not exactly sure if it is really this line's issue that caused the exception)
• package.json the leaflet-vector-tile-layer package version probably needs to be updated if a version with setUrl is released

One issue that I notice: After the url is updated using my PR on VectorTileLayer repo, the layer itself will not update immediately. Only when user zoom in or out, or perhaps toggle the vector tile layer off and on, the new tiles will be sent to clientside and thus the layer will reflect the new url. To make the url change appears immediately, I use the following chained callback functions:

@callback(
    [Output("building","url"),Output("buildingDisplayToggle","checked")],
    [Input("homepagemap","zoom")],
    State("building","url"),
    prevent_initial_call=True,
)
def updatebuildinglayer(zoomlevel,current_url):
    if zoomlevel > 16:
        expected_url = "<Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?properties=res,comind,geometry&filter-lang=cql2-text&filter=res IS NULL AND comind IS NULL"
        if expected_url != current_url:
            print(expected_url)
            return [expected_url,False]
    else:
        expected_url = "<Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?properties=res,comind,geometry"
        if expected_url != current_url:
            print(expected_url)
            return [expected_url,False]
    
    return dash.no_update

clientside_callback(
    """
    function(_buildingurl) {
        return true;
    }
    """,
    Output("buildingDisplayToggle","checked",allow_duplicate=True),
    Input("building","url"),
    prevent_initial_call=True
)

[zifanw9]

Hi @emilhe , I created a new PR that adds an URL update method for feature filtering use case. I didn't get my PR on Leaflet.VectorTileLayer repo approved, but I implement another way for doing it after discussion with Joachim. I provide more details in the PR. Can you please see if a new release can be made based on that PR when you have time? Thank you!

[fallspinach]

This VectorTileLayer feature is great! I've been waiting for this! I tried the example above and it works well but couldn't get my own data shown up on the map. I suspect the PBF file I have isn't in the right format, for example:

https://cw3e.ucsd.edu/wrf_hydro/merit_rivers/0/0/0.pbf

Does anyone know if this tile data is created properly? If not, what tool should I use to generate the correct data? Thanks so much! BTW, the data shows up fine in QGIS.

Edit: I'm using basically the same code from the example except for the tile url and zoom level:

import dash_leaflet as dl
from dash import Dash

app = Dash()
app.layout = dl.Map(
    [
        dl.TileLayer(),
        dl.VectorTileLayer(url="https://cw3e.ucsd.edu/wrf_hydro/merit_rivers/{z}/{x}/{y}.pbf", maxDetailZoom=3, style={}),
    ],
    center=[56, 10],
    zoom=1,
    style={"height": "50vh"},
)

if __name__ == "__main__":
    app.run_server(debug=True)

[Elliott-Rose-BSC]

I would like to add another request that this is done for DL. I am implementing vector tiles in a dash app now, and without the leaflet support I'm having to do all kinds of work arounds.

[aliciaoberholzer]

Hi @emilhe & the dash-leaflet community,

I'd like to add my voice to those requesting the VectorTileLayer component be finalized and merged into the main release. I'm currently working on visualizing large geospatial datasets where the performance advantages of vector tiles make a significant difference to end-users.

I'm testing Dash as a replacement for PowerBI, but am encountering limitations with the current workarounds for visualizing vector tiles. Below is a screenshot of the type of PowerBI visualizations I'm trying to replicate in Dash - showing gender norm data in Nigeria down to the ward level. Currently, even using simplified GeoJSON causes significant performance issues.

I would greatly benefit from having this feature officially supported in the main package and would be happy to help test or provide feedback on the implementation if needed.

Thank you for considering this request and for all your work on dash-leaflet

[emilhe]

@aliciaoberholzer can you eloborate on how close the current draft implementation is to matching your use case? What additional features / functionality is needed?

[emilhe]

@fallspinach do you have any Leaflet example(s) where the visualization is working?

EDIT: I just tried your example. The tiles don't load due to a CORS erro (i.e. the browser blocks the request for security reasons). You'll probably need to adjust the server to add proper headers (or use a proxy). Since QGIS isn't a browser, CORS is not an issue, so it makes sense that it "just works".

[aliciaoberholzer]

Hey @emilhe - thank you for your response! I'll work on writing something up for you about my current work use case!

[fallspinach]

@emilhe Thanks so much for your help and sorry for my slow response. I removed the tile data earlier when I was trying something else. I just put the tiles back on the server and created a leaflet javacript example here:

MERIT Rivers Leaflet Example (using VectorGrid)

The above example will show the vector tiles up to zoom 5. So far the leaflet javascript works with these tiles but dash-leaflet doesn't. I tried another way to display the tiles here without using dash-leaflet:

GRADES-hydroDL on MERIT-Basins Rivers (over Terrains)

But of course, it'll be much much better if I can make dash-leaflet work with these tiles. Can you try my example again and see what error messages you see? Or just let me know where to find the error messages and I'll debug it. Thanks so much!