NASA-AMMOS
Plugins: Add support for other tiled formats
2D Formats
- [ ] Protomaps
- https://protomaps.com/
- 2d vectorized data with text / metadata information
- [ ] Overture Maps Data
- https://overturemaps.org/
- [ ] XYZ MVT Data
- OpenStreetMap also provides vector tiles in the MVT format.
- Another vector format? here
- [ ] Cloud Optimized GeoTiff
- Tiled TIFF file with different levels of detail embedded
- https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
- [ ] 3D OpenStreetMap Data
- https://wiki.openstreetmap.org/wiki/3D
- [ ] Mapbox Tilejson Format
- https://github.com/mapbox/tilejson-spec
- [ ] LERC
- Definition: https://github.com/Esri/lerc (including JS parser)
- Files: https://services.arcgisonline.com/arcgis/rest/services/WorldElevation3D
- structured as WMTS
- [ ] KML / KMZ
- https://en.wikipedia.org/wiki/Keyhole_Markup_Language
- [ ] Other
- ~Ellipsoid projection~
- Repeat tiles on X and Y
- [x] WMS
- #1299
- #1302
- [x] GeoJSON
- #1305
- [x] WMTS Capabilities
- Support the WMTS Capabilities format to infer tile mtrices, etc without explicit user specification or url templating. See sample here and GIBS capabilities here.
- Add a parser to enumerate all the layer information and options / dimensions.
- Allow for passing the list directly into the WMTSLayer for construction or passing the capabilities url in directly and specifying the keys to use.
- https://services.arcgisonline.com/arcgis/rest/services
- [x] WMTS tiles for rendering 2d map data
- Can support planar or ellipsoid projection
- https://api.oneatlas.airbus.com/guides/oneatlas-basemap/g-oab-wmts-basic/
- https://opengeospatial.github.io/e-learning/wmts/text/main.html
- Seems to be TMS but flipped on Y
- Cesium (and many other libraries) seem to avoid the capabilities file and just require passing in the url parameters (see cesium's implementation here).
- It seems Cesium also recommends the "OpenLayers" parser here or here. ~Or a leaflet integration capability here.~
- WMTS works currently with XYZ format template but would be good to support this more officially / directly. Eg class or string interpolation arguments (matrix set, time).
- [x] TMS data
- Supported in Cesium Ion "imagery"
- https://cesium.com/learn/3d-tiling/tiler-data-formats/
- [x] XYZ data
- Max zoom level is 19
- https://wiki.openstreetmap.org/wiki/Tiles
- https://gis.stackexchange.com/questions/447421/convert-a-point-on-a-flat-2d-web-mercator-map-image-to-a-coordinate
- [x] Deep Zoom Images for rendering high resolution images
- #945
- If needed could add support for Collections, DisplayRects
DEM Formats
- [ ] TerrainDEM
- https://docs.mapbox.com/data/tilesets/reference/mapbox-terrain-dem-v1/
- [ ] AWS Open Terrain Tiles
- https://registry.opendata.aws/terrain-tiles/
- [x] QuantizedMesh tiles for rendering ion terrain
- https://github.com/CesiumGS/quantized-mesh
- Provide an option to construct a solid volume CSG support.
3D Formats
- [ ] PoTree
- Point cloud hierarchy structure
- https://github.com/cpj001/potree-develop/blob/master/docs/potree-file-format.md
- [ ] COPC
- COG-inspired point cloud format
- https://copc.io/
- [ ] I3S
- Generalized 3d hierarchy system similar to 3d tiles.
- https://www.ogc.org/publications/standard/i3s/
Coordinate Reference Systems
- EPSG:3031, used for Antarctic data sets. Some ArcGIS data uses this.
- EPSG:21781, used for Swiss data sets.
Data Sources
- ESA Data: https://esa-worldcover.org/en/data-access
- Terrascope: https://docs.terrascope.be/Developers/WebServices/OGC/WMTSv2.html
- Mapbox
- Jonni Walker's data sets (see here)
- OpenStreetMap tiles (here)
- ArcGIS data sets (here)
- NASA GIBs data sets (here)
- Cesium Ion
- https://geotiles.citg.tudelft.nl/
- Protomaps (here)
- OpenDroneMap.org
- Esri data format - may require custom vector data format
- https://www.arcgis.com/home/item.html?id=33064a20de0c48d2bb61efa8faca93a8
- https://www.arcgis.com/home/group.html?id=f2d67dbc6a764031969b2e3a891fea90#overview
- Swiss EPSG data sets
- 3D tiles of vectorized location names
- WMTS in EPSG:21781 (Swiss projection), in particular SWISSIMAGE (swiss aerial imagery)
- Dutch Data Sets (see #1323)
- OpenBasisKaart: https://www.openbasiskaart.nl/mapcache/wmts/?SERVICE=WMTS&REQUEST=GETCAPABILITIES
- PDOK Luchtfoto: https://service.pdok.nl/hwh/luchtfotorgb/wmts/v1_0?request=GetCapabilities&service=WMTS
- PDOK Achtergrondkaart: https://service.pdok.nl/brt/achtergrondkaart/wmts/v2_0?request=GetCapabilities&service=WMTS
- PDOK Kadaster KadastraleKaart: https://service.pdok.nl/kadaster/kadastralekaart/wmts/v5_0?request=GetCapabilities&service=WMTS
- PDOK Kadaster BGT: https://service.pdok.nl/lv/bgt/wmts/v1_0?request=GetCapabilities&service=WMTS
Authentication APIs
- ~bing maps~
- here maps
- mapbox
- maptiler
- openmaptiles
Regarding OSM tiles, they indeed introduced a few months ago a production vector tiles endpoint which can be styled client-side: eg tilekiln-shortbread-demo from Paul Norman, and the official OSM forum thread, where style can be live-edited via TileJSON editors like Maplibre Maputnik, mappable or maptiler: https://vector.openstreetmap.org/shortbread_v1/{z}/{x}/{y}.mvt.
Regarding other types of Tiled 3D data, I just wanted to quote that loaders-gl has a section dedicated to tiled-3d-data loaders here, with support already live for OGC 3D Tiles and I3S. In their v5.0 roadmap, they aim for supporting other tiled formats like
Also for what it is worth, I dreamt once about a TiTiler-like middleware but for 3d, but it did not gather attention (TiTiler is given an endpoint to a COG Cloud Optimized Geotiffs, and returns a live TMS endpoint with on-the-fly tile selection, mosaicking, merging and reprojection), to dissociate data format from consuming end-user application - zero-traction post and associated quick discussion in loader-gl repo.
Thanks for the info! Getting Potree working would be great considering some of the discussions in #912, as well. I'm less familiar with COPC - is this a widely used format? It looks fairly new, still. I'll have to take a look at the I3S format a bit more, too. The spec looked a bit difficult to understand and I'm not sure if it's really designed for web use.
TileJSON editors like Maplibre Maputnik, mappable or maptiler:
https://vector.openstreetmap.org/shortbread_v1/{z}/{x}/{y}.mvt
Do you have a link to the spec that OSM uses for tiling? Is it something similar (or the same as) WMTS?
TiTiler is given an endpoint to a COG Cloud Optimized Geotiffs
Supporting Cloud Optimized Geotiffs directly should be possible, as well, with no middleware (looks like there are embedded LoDs). Sounds like something like TiTiler is good for enabling support in applications that only support WMTS but some of the these tiling formats are close enough to be on-the-fly converted client side.
From my point of view, Potree succeeded because it was one of the first, easiest way to tile massive pointclouds for web viewing, with a viewer and library with the right amount of tooling for measurement taking, clipping, POIs, and easiness to deploy for non tech-savvy users. So a lot of public and private organizations are using it, which result in a pretty large collection of datasets in that spec. The older v1 format stored the hierarchy tiles as individual files which were named by their index, which was pretty heavy on the filesystem during transfers, compression etc. Markus introduced the v2 format a few years back to alleviate these, resulting in a single file tiled pointcloud.
COG is simply a geotiff but with pyramid overviews (downsampled version of the file) and block memory-contiguity (for faster retrieval of a subset of the data), standardized. Titiler indeed takes advantage of that internal tiles structure to decide which portions of the file to read, retrieve them, reproject and merge on the fly, with gdal wizardy - I even got TiTiler to ingest not only COGs, but also other TMS like in this discussion.
COPC has been created by the team at hobu (folks behind eg PDAL), which has been dealing with massive pointclouds from a long time, eg 3DEP, the lidar coverage of the US as open-data. My understanding is they first created Entwine as an open spec, similar to potree, with extensibility in mind, but then got the inspiration from COG to create COPC. They extended the LAS file format, and describe how tiles should be described in header. If an app does not know COPC, it can just parse the file as a standard las. COPC os getting used more and more by national mapping agencies to publish their Lidar coverage, especially in US/Europe. PDAL is usually used to convert a standard pointcloud to COPC.
From my understanding, I3S never got traction outside of the ESRI ecosystem, which inceptionned it, especially in comparison to 3D-Tiles, which was pushed more by Cesium - especially with all the ecosystem of plugins for 3D render engines, now on Unreal, Unity, O3DE, Omniverse in addition to the web libs.
The /{z}/{x}/{y} used by OSM is indeed the standard TMS or XYZ tiling scheme (I often find myself looking back at this ref), in web mercator EPSG:3857 - with sometimes a Y-inversion with Google tiling, y_google = 2**zoom - y_tms - 1.
Hope, this all makes sense!
Great, thanks! This is all helpful information.
The /{z}/{x}/{y} used by OSM is indeed the standard TMS or XYZ tiling scheme (I often find myself looking back at this ref),
Can you explain the difference between TMS and WMTS, then? I've tried looking it up before but it wasn't super clear to me. A simple explainer would help.
TMS and XYZ are endpoint naming schemes/conventions, where tile indices (zoom-level=z, column=x and row=y) are usually represented surrounded by brackets. Any standard file server can act as a TMS server as long as tile files are stored with the described hierarchy directories, eg upper level directory for zoom, which each contain one folder per column, which itself contain one file per row - eg here satellite https://1.aerial.maps.ls.hereapi.com/maptile/2.1/maptile/newest/satellite.day/{z}/{x}/{y}/512/jpg. Other url examples for TMS include google satellite https://www.google.cn/maps/vt?lyrs=s@189&gl=cn&x={x}&y={y}&z={z}, where url-params are used rather than hierarchical directory.
WMS/WMTS is a more complex server specification, where the server has to serve raster tiles via the GetTile endpoint, but also other endpoints, like GetCapabilities which describe what the server can produce and CRS it handle. URLs are usually in the form https://wxs.ign.fr/essentiels/geoportail/wmts?SERVICE=WMTS&REQUEST=GetTile &VERSION=1.0.0&LAYER=ORTHOIMAGERY.ORTHOPHOTOS&TILEMATRIXSET=PM &TILEMATRIX={z}&TILECOL={x}&TILEROW={x}&STYLE=normal&FORMAT=image/jpeg. WMTS is a WMS server with tiles cached.
You can see a collection of WMS vs TMS endpoints on NextGIS QMS, with satellite endpoints for google, here, esri, bing (based on quadkey rather than xyz) etc.
Quick note regarding 2.5D DTM/DSM formats (quantized-mesh is one of them rather than real 3D like the others, COPC, potree, 3dtiles, I3S etc): there are two existing standards for encoding height in RGB 3-channel data image, with a pretty good height accuracy. They are usually served as tiles, via TMS, WMTS endpoints etc.
- MapTiler/Mapbox TerrainRGB:
elevation = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1), mapbox serves their terrain data following this standard - Terrarium,
elevation = (R * 256 + G + B / 256) - 32768eg Amazon Open Data Terrain Tileshttps://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png.
I recently discovered within Titiler (middleware initially meant to ingest COGs, Cloud Optimized Geotiffs, and serve on-the-fly a TMS endpoint) that algorithms can be applied to the data, and therefore it is possible to encode/decode standard-height/terrarium/terrainrgb - see this discussion and PR fixing terrain-rgb encoding. Note it is also capable of computing hillshade or slope also on the fly, and not only ingest DSM COGs but also TMS for height data.
Following discussion in #1264, I would like to share the following currently unsupported datasets. They come from Swiss geospatial information system (docs available here).
- 3D tiles of vectorized location names (see #814)
- WMTS in EPSG:21781 (Swiss projection), in particular SWISSIMAGE (swiss aerial imagery) which is only available at zoom level 28 (10cm resolution) in this projection. Otherwise EPSG:3857 allows only zoom level 20 max.
- WMTS in EPSG:21781 (Swiss projection), in particular SWISSIMAGE (swiss aerial imagery) which is only available at zoom level 28 (10cm resolution) in this projection. Otherwise EPSG:3857 allows only zoom level 20 max.
@gkjohnson Following your reminder about EPSG projections in https://github.com/NASA-AMMOS/3DTilesRendererJS/issues/1299#issuecomment-3242234340, I would like to request from you high level guidelines in order to open a PR to support other EPSG projections, if you are open and ready to have this improvement implemented now.
I would like to request from you high level guidelines in order to open a PR to support other EPSG projections, if you are open and ready to have this improvement implemented now.
Thanks @xawill! I think a dedicated issue would be the best place to start. Right now the "CRSs" are really just used to determine and apply any projection distortion for the sake of rendering it correctly while data ranges are always normalized to cartographic coordinate in the lon / lat order. WMS is the first time it's actually be necessary to transform back into one of these CRSs for fetching and rending a data set.
From looking at the Swiss EPSG:21781 CRS definition it seems like it's using the same kind of projection as the EPSG:4326 WGS84 CRS just defining a subframe. So what I need to understand is how this actually impacts image tiles that are referencing the Swiss frame. For example: does it change where the tile set origin is defined? Is it just defining the "contentful" tiles in the map while the tiling scheme origin remains the same? Is it different per map format? Once we understand some of this I think it will become more clear how to integrate some more CRSs.
With the recent addition of GeoJSON support (#1305), it might be interesting to consider support for 'tiled' vector formats like WFS (similar to WMS it does not serve data in pre-generated tiles but based on a user specified region). In fact this already somewhat works if you pass a WFS request that returns geojson as a url to a GeoJSONOverlay. There is a lot of data in WFS.
Would that fit in this library?
In fact this already somewhat works if you pass a WFS request that returns geojson as a url to a GeoJSONOverlay. There is a lot of data in WFS.
I'm unfamiliar with WFS but if WFS already supports retruning data as GeoJSON is there a need to do anything to support it explicitly? Using the URL provided it looks like it already works as-is:
Can you give a brief overview of WFS? I'm trying to avoid adding support for source format just because it exists but if there are some that are commonly used, have data locked into them, or provide unique functionality then it may be worth considering.
It's pretty impressive seeing the great progress of 3DTilesRendererJS towards supporting more and more geospatial formats - tiled like TMS/WMS/WMTS or not like GeoJSON. Congrats to you and new contributors on this effort!
I'm wondering whether at some point it would not be duplicating the efforts too much to reimplement support for other formats vs adding bindings to a library like visgl/loaders.gl, which already implements native JS support for most Geospatial Formats. Their example sidebar section showases the loaders, and categorizes them as below. This lib is the foundation behind deckl and keplergl (originally uber engineering, spin off unfolded studio acquired by FSQ studio) and they also have some open discussions for 3d tiled format not yet part of the implementation - see the list below.
-
Not yet implemented 3D-tiled:
-
Geospatial Table Formats
- GeoJSON
- GeoPackage
- GeoParquet
- KML
- others like FlatGeobuf, GeoArrow, Shapefile, GPX, TCX
-
Geospatial Tile Formats
Edit 2: quick note that since 3DTilesRendererJS supports TMS/WMS, any GDAL-compatible raster tiled format or endpoint (especially COG, could have been WMS/WMTS when you only supported TMS/XYZ) could be served by developmentseed/titiler middleware, see example discussions [1], [2}
EDIT: the WFS standard is like WMS but for vector features. Basically, if you have multi-millions of features that you do not want to send in one batch to the client, a WFS server implements clustering mechanisms so tiles can be requested at lower zooms. Actually closer to an MVT endpoint - which OSM recently published OSM vector tiles based on that spec. Its cloud-native geospatial equivalent is pmtiles, which is one file with hierarchy table inscribed in header, and can be range-requested for specific viewport/subsamples.
I'm unfamiliar with WFS but if WFS already supports retruning data as GeoJSON is there a need to do anything to support it explicitly? Using the URL provided it looks like it already works as-is.
Indeed I did the same, and this already works if you only want one specific region. What could be added on top of this is a mechanism similar to how WMS is implemented with a Capabilities loader (using a GetCapabilities request) and something to dynamically load features based on where the viewport is (using GetFeature request with a bbox).
I will say that WFS is not the most performant. Loading more than a few hundred features can be time consuming, and there is no hierarchical LOD scheme. In that sense MVT/PMTiles or MLT (advertised as a better MVT) are more interesting.
Sorry for the delay - just coming back around to being able to work on some stuff again.
@jo-chemla
I'm wondering whether at some point it would not be duplicating the efforts too much to reimplement support for other formats vs adding bindings to a library like visgl/loaders.gl
I'm familiar with the project and it looks great but from the docs it doesn't look like they would have saved a whole lot for the the recent features. The "hard" part of these formats is understanding the (sometimes convoluted) tiling schemes that are defined, fetching / drawing the data to textures, expanding the data into a 3d tiles hierarchy, etc. None of which seem to be handled by the loaders. Most of the work for WMS, WMTS, etc has been understanding how to interpret the data from the source format documentation and in this case these loaders will really just add an extra layer of indirection to understanding that, for me at least. It's worth considering if the visgl may be worth using in other formats when they come up, though.
With #1309 I'd like to remove the dependency on the returned result format for WMS and WMTS when instantiating overlays, though, so it should be easier for users to use the visgl loaders if they so choose and maybe someday we can remove the capabilities loaders in this project entirely and direct users to use another projects implementation.
@Ylannl
something to dynamically load features based on where the viewport is (using GetFeature request with a bbox).
Can you explain this a bit more? Do you mean something like being able to say "load X feature type within this region only"? Or "only load this feature after X level of detail is reached"?
Can you explain this a bit more? Do you mean something like being able to say "load X feature type within this region only"? Or "only load this feature after X level of detail is reached"?
It would have to be both, because 1) with WFS you can only retrieve features by bounding box and 2) it only really makes sense to load after X level of detail (ie fairly small bounding box), because WFS is not designed to handle requests that would return a large number of features. What typically happens is that the server has a limit on the number of features it will return per request and when the request bounding box becomes too large it will just randomly omit features, which can be confusing to the user.
I'm trying to avoid adding support for source format just because it exists but if there are some that are commonly used, have data locked into them, or provide unique functionality then it may be worth considering.
I completely understand, and as mentioned above, more modern vector tile formats exist that are technically superior to WFS. So I would definitely prioritise those over WFS.
Thanks for your time and consideration!
Thanks for getting back regarding loadersgl and the visgl ecosystem, makes complete sense + great to see they were already on your radar.
Set aside classic file-based geospatial raster/vector formats, my high-level understanding for loadersgl 2D-tiled and 3D-tiled loaders was that they were designed so that the underlying raster/vector/3D hierarchical structure was mapped to a unified abstraction. This abstraction makes it easier (again my pov) for the underlying mapping library (whatever it is for deckgl, maplibre etc) to decode the dataset data/metadata, decide which chunks to load, fetch, reproject into the underlying map CRS, and render them. I did not read the codebase so I'm probably wrong regarding the loadersgl library also handling and exposing the tiling logic, 2D/3D LOD refinement strategies or CRS reprojection - @ibgreen might have useful insights there, sorry for the ping, might be a great addition!
Also, completely true that you would need to map/convert the bits abstracted and exposed by the loadersgl objects into objects you could re-use within the threejs set of classes, textures, polylines, to be overlaid onto a 3D-tiles tileset, ellipsoid etc.
Motivation for additional terrain sources
In case it's useful, I just gathered several 2.5D TerrainRGB and Terrarium resources, existing terrain tile providers, and references in a similar feature request made on the CesiumJS repo here https://github.com/CesiumGS/cesium/issues/13035#issue-3628418160 - see the interesting bits quoted below.
I've used these for a POC maplibre app to compare terrain tiles providers (both 2.5D terrain, as well as viz-modes hillshade, color-relief & contours), that should be released soon, see screencast demo.
Allowing such elevation/terrain sources to be rendered via 3DTilesRendererJS would offer a wider range of material customization, like:
- raster source reprojection of sources already handled by the lib (TMS, WMTS, WMS etc) or upcoming tiled formats
- proper lighting via standard threejs materials via point (or other) light sources
- postprocessing via eg takram-design-engineering/three-geospatial/r3f/pmndrs/postprocessing
- custom shader for slope, aspect, or any normal-derived attributes, via eg postprocessing passes - for more context, additional feature requested to maplibre here https://github.com/maplibre/maplibre-style-spec/issues/1374.
TerrainRGB/Terrarium server endpoints
[...] to support additional terrain representations - similar to how libs like maplibre, mapbox do it - especially the following elevation RGB encodings:
TerrainRGBby Mapbox/MapTiler:height = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1)Terrariumby Mapzen:height = (R * 256 + G + B / 256) - 32768- a bit more convoluted: DEM COG (or DSM/DTM, Digital Elevation Model) support added via geomatico/maplibre-cog-protocol see example
The above 2 specifications are similar to heightmap-1.0 in that the endpoints serve TMS templated URLs like
https://tiles.mapterhorn.com/{z}/{x}/{y}.webpin EPSG:3857 WebMercator tiles, usually 256² raster tiles where RGB triplets encode elevation via the above formulas/equations. Some providers also provide a TileJSON endpoint, like Mapterhornhttps://tiles.mapterhorn.com/tilejson.jsonExample providers serving terrain-rgb or terrarium encoded terrain
- Mapterhorn open-data
https://tiles.mapterhorn.com/{z}/{x}/{y}.webp- AWS Terrain Tiles open-data
https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png- MapTiler terrain via API-key
https://api.maptiler.com/tiles/terrain-rgb-v2/{z}/{x}/{y}.webp?key={KEY}- Mapbox Terrain-DEM v1 via API-key
https://api.mapbox.com/v4/mapbox.terrain-rgb/{z}/{x}/{y}.png?access_token={KEY}Note 1: The libs mentioned (maplibre, mapbox etc) usually handle these elevation datasets both as
- terrain, a 2.5D displacement of the globe or tilted plane surface,
- as well as for other visualization modes (example layer types for maplibre: hillshade, color-relief, contours etc)
The XYZTilesPlugin should be pretty straightforward to extend too support something like TerrainRGB or Terrarium files. It's really just a matter of reading the texture data to the CPU and modifying the geometry to account for the heights. I'm open to PRs for these formats if anyone wants to give it a go. I can give any direction in a dedicated issue or PR if it's needed.
Thanks for getting back, I just opened a dedicated issue to track progress and get directions regarding Terrarium/TerrainRGB support, here https://github.com/NASA-AMMOS/3DTilesRendererJS/issues/1370