Specifications for vector tiles loading? (both .pbf and metadata.json)

[TacticalGator]

I cannot find any proper documentation anywhere regarding the specifications required for vector tiles loading (both .pbf and metadata.json).

I tried loading my own vector tiles & metadata that include a timestamp in ISO8601, but kepler.gl parses it as strings and it's not showing up in the filter.

Can someone point me to the right direction?

Edit 2: I tried converting ISO8601 to unix timestamp and declare it as "number" on metadata.json. Kepler.gl parsed it as float and made it an available "field" in filter now. "time", "date", "datetime" types don't seem to be available for vector times & metadata loading. Can somebody help?

[igorDykhta]

Hi @TacticalGator, It looks like this isn’t supported at the moment. Tippecanoe (and the MVT spec) don’t have a native DATE/TIME type, and type handling currently happens before we receive any data, when we process the metadata.

I’ll add this as a feature request.

There’s a fallback in place right now: when proper fields are missing from the metadata, the inner logic tries to fetch (I think) the central tile based on the metadata bounds and then infers the types from that tile. Apparently this won't work in many cases.

[TacticalGator]

Hey @igorDykhta ! Thank you so much for taking a look into this.

There appear to be several implementations of vector tile "specs", most of which are wrapped in a binary format like protobuf. This got me real confused as to how I could implement one in order to fit into the Kepler.gl's ecosystem.

There are so many tool our there, but Tippecanoe being the sole vector tile example that I could find: https://github.com/keplergl/kepler.gl-data/tree/master/mvt_population

so I attempted to reconstruct it using a Python script.

In addition, only "float" data type is showing up and configurable in the filter. Everything else, boolean, int, string, etc are missing in the filter. I am not entirely sure if this is an error on my end though.

[igorDykhta]

@TacticalGator string filters with tilesets are definitely not supported. int - hm, should be supported.

[igorDykhta]

https://cdn.unfolded.ai/indigo/hexify_v5/{z}/{x}/{y}.pbf https://r2-public.protomaps.com/protomaps-sample-datasets/cb_2018_us_zcta510_500k.pmtiles https://api.mapbox.com/v4/mapbox.mapbox-streets-v8/{z}/{x}/{y}.mvt?access_token=XXX https://r2-public.protomaps.com/protomaps-sample-datasets/nz-buildings-v3.pmtiles https://fsq-os-places-us-east-1.s3.us-east-1.amazonaws.com/release/vector-tiles/latest/fsq-os-places.pmtiles

[TacticalGator]

@igorDykhta Wow so quick! Thank you so much for links! They are exactly what I needed to understand vector tiles in kepler.gl. I will test it again with int

[igorDykhta]

Bool and int should definitely work. Seems like a bug.

[igorDykhta]

If you can generate some interesting examples, I can add them to the docs.

[TacticalGator]

Bool and int should definitely work. Seems like a bug.

@igorDykhta So the metadata for the first example link that you shared, has both integer(croptype) and float(season_means) attributes as "number" as well as a boolean(`till_class) attribute.

On the version 3.2.0, integer(croptype) is showing up as float while boolean is correctly shown.

If you can generate some interesting examples, I can add them to the docs.

I got some vector tiles that I've been testing locally. Not entirely sure how I could provide you some examples.

[TacticalGator]

Hi @TacticalGator, It looks like this isn’t supported at the moment. Tippecanoe (and the MVT spec) don’t have a native DATE/TIME type, and type handling currently happens before we receive any data, when we process the metadata.

I’ll add this as a feature request.

There’s a fallback in place right now: when proper fields are missing from the metadata, the inner logic tries to fetch (I think) the central tile based on the metadata bounds and then infers the types from that tile. Apparently this won't work in many cases.

Hello @igorDykhta Any progress on this feature?

[igorDykhta]

@TacticalGator Not really. What do you think is the better way to handle this? Add non-standard type hints in the metadata, or add an optional UI to define change types?

[TacticalGator]

@igorDykhta Adding non-standard type hints is not really practiced with MVP and would become kepler.gl-specific if implemented. I think that the latter is much more flexible and would give users more control.

Besides, most temporal data will come in either ISO8601, RFC3389, or the Unix timestamp format. ISO8601 and RFC3389 seem pretty straight forward have Kepler.gl to "detect", whereas the Unix timestamp format might be challenging it would need range checks or context/heuristics, etc.

[igorDykhta]

Well, it won't be an automatic detection; instead, the user will need to provide additional information about the types, as we don't have sample data to infer the types from.

[TacticalGator]

@igorDykhta Sounds good. I really wish that I had enough React and Redux knowledge to hop in.

There is also another feature idea that I had in mind. A unified time slider that users can add data sources to. I've been trying to hack my way around using LLMs but not much avail.

Please let me know once you implement this. I would love to test it out.

[igorDykhta]

@TacticalGator Time sync is implemented. You can add several datasets with timestamp or dates(?). Then in filter you filter by that timestamp. And you should be able to add + synced datasets.

[TacticalGator]

@igorDykhta Thanks for letting me know. I've only been working on vector tiles that I didn't even realize that it was there.

For my use case specifically, I have to solely rely on vector tiles as individual data source yields about half a million features and there are currently over 20+ data sources. I'm limiting the amount of features via my API backend, so that gives some breathing room. Additionally, none of them are static. There are all NRT ± 10 seconds up to 3 hours.

Once the time type for vector tiles gets implemented, everything will works perfectly. I can't wait to see!

[TacticalGator]

Hi, @igorDykhta I hope you are doing well. I have another question.

During testing, I've been manually loading and configuring filters & layers for vector tiles datasets on the web UI, but this has become incredibly tedious and time-consuming as my backend grew.

• addDataToMap doesn't seem to be supporting vector tiles.

• I also figured out a way to save the layer & filter config using KeplerGlSchema.getConfigToSave.

• There seems to be a config JSON file that can be loaded onto Kepler.gl, but there are no instructions and also no vector tiles; https://github.com/keplergl/kepler.gl-data/blob/master/samples.json

How could I load vector tiles datasets programmatically? Would you be able to provide a solution for this?

[TacticalGator]

Hi, @igorDykhta I hope you are doing well. I have another question.

During testing, I've been manually loading and configuring filters & layers for vector tiles datasets on the web UI, but this has become incredibly tedious and time-consuming as my backend grew.

* `addDataToMap` doesn't seem to be supporting vector tiles.

* I also figured out a way to save the layer & filter config using `KeplerGlSchema.getConfigToSave`.

* There seems to be a config JSON file that can be loaded onto Kepler.gl, but there are no instructions and also no vector tiles;  https://github.com/keplergl/kepler.gl-data/blob/master/samples.json

How could I load vector tiles datasets programmatically? Would you be able to provide a solution for this?

RESOLVED via dispatching updatevisdata

[TacticalGator]

Hi @TacticalGator, It looks like this isn’t supported at the moment. Tippecanoe (and the MVT spec) don’t have a native DATE/TIME type, and type handling currently happens before we receive any data, when we process the metadata.

I’ll add this as a feature request.

There’s a fallback in place right now: when proper fields are missing from the metadata, the inner logic tries to fetch (I think) the central tile based on the metadata bounds and then infers the types from that tile. Apparently this won't work in many cases.

Hi @igorDykhta, Any update on this feature so far?

I made some changes to have Kepler.gl recognize timestamp internally. The Time Playback feature also partially works while not showing the y-axis. The "sync" with other datasets also works.

Please have a look at changes made in the src directory: https://github.com/keplergl/kepler.gl/compare/master...TacticalGator:kepler.gl:master

[igorDykhta]

Nothing so far