PMTiles OpenLayers plugin

Feb 24 '21 12:02 bdon

I just discovered this project and official support for Openlayers would be awesome 👍 Btw here is an open issue that discusses support for alternative tile sources like yours: https://github.com/openlayers/openlayers/pull/12008#issue-570057745 So maybe you could contribute to those efforts?!

Apr 26 '21 11:04 fl0cke

Hi, agreed this would be great! The major blocker is for the OpenLayers API to allow asynchronous tile creation, which I was not able to figure out based on looking at the code or any plugins. If anyone has any tips or can show an example I would be happy to continue on this path.

Apr 26 '21 13:04 bdon

Hey, here is the code i'm currently using (it's still very fresh):

import { TileImage } from 'ol/source'
import State from 'ol/source/State'

import { extentFromProjection } from 'ol/tilegrid'
import TileGrid from 'ol/tilegrid/TileGrid'
import { getHeight, getWidth } from 'ol/extent'
import TileState from 'ol/TileState'

function shift(number, shift) {
  return number * Math.pow(2, shift)
}

function getUint24(dataview, pos) {
  return (
    shift(dataview.getUint16(pos + 1, true), 8) + dataview.getUint8(pos, true)
  )
}

function getUint48(dataview, pos) {
  return (
    shift(dataview.getUint32(pos + 2, true), 16) + dataview.getUint16(pos, true)
  )
}

export default class PMTiles extends TileImage {
  constructor(options) {
    const extent = options.extent || extentFromProjection(options.projection)

    const tierSizeInTiles = []
    const width = getWidth(extent)
    const height = getHeight(extent)
    const tileSize = options.tileSize
    let tileSizeInTier = tileSize

    while (width > tileSizeInTier || height > tileSizeInTier) {
      tierSizeInTiles.push([
        Math.ceil(width / tileSizeInTier),
        Math.ceil(height / tileSizeInTier),
      ])
      tileSizeInTier += tileSizeInTier
    }

    tierSizeInTiles.push([1, 1])
    tierSizeInTiles.reverse()

    const resolutions = [1]
    for (let i = 1, ii = tierSizeInTiles.length; i < ii; i++) {
      resolutions.push(1 << i)
    }
    resolutions.reverse()

    const tileGrid = new TileGrid({
      tileSize,
      extent,
      resolutions,
    })

    super({
      state: State.LOADING,
      attributions: options.attributions,
      cacheSize: options.cacheSize,
      crossOrigin: options.crossOrigin,
      imageSmoothing: options.imageSmoothing,
      opaque: options.opaque,
      projection: options.projection,
      reprojectionErrorThreshold: options.reprojectionErrorThreshold,
      tileGrid,
      tilePixelRatio: options.tilePixelRatio,
      wrapX: options.wrapX !== undefined ? options.wrapX : true,
      transition: options.transition,
      attributionsCollapsible: options.attributionsCollapsible,
      zDirection: options.zDirection,
    })

    this.maxResolution = resolutions[0]

    this.url = options.url

    this.tileLoadFunction = this.loadTile.bind(this)

    this.init()
  }

  async init() {
    try {
      const response = await fetch(this.url, {
        headers: { Range: 'bytes=0-511999' },
      })
      const data = await response.arrayBuffer()
      const header = this.parseHeader(new DataView(data, 0, 10))
      this.tileMap = this.buildTileMap(
        new DataView(data, 10 + header.jsonSize, 17 * header.rootEntries)
      )
      this.setState(State.READY)
    } catch (e) {
      this.setState(State.ERROR)
    }
  }

  tileUrlFunction(tileCoord, pixelRatio, projection) {
    return this.url + `/${tileCoord[0]}-${tileCoord[2]}-${tileCoord[1]}`
  }

  parseHeader(dataView) {
    const magic = dataView.getUint16(0, true)
    if (magic !== 19792) {
      throw new Error('Magic number does not match')
    }
    const version = dataView.getUint16(2, true)
    const jsonSize = dataView.getUint32(4, true)
    const rootEntries = dataView.getUint16(8, true)
    return { version, jsonSize, rootEntries }
  }

  buildTileMap(dataView) {
    const m = new Map()
    for (let i = 0; i < dataView.byteLength; i += 17) {
      const zRaw = dataView.getUint8(i)
      const z = zRaw & 127
      const isDir = zRaw >> 7
      const x = getUint24(dataView, i + 1)
      const y = getUint24(dataView, i + 4)
      const offset = getUint48(dataView, i + 7)
      const length = dataView.getUint32(i + 13, true)
      m.set(z + '-' + y + '-' + x, { offset, length, isDir })
    }
    return m
  }

  async loadTile(tile, src) {
    const key = src.substring(src.lastIndexOf('/') + 1)
    const tileMeta = this.tileMap.get(key)
    try {
      const response = await fetch(this.url, {
        headers: {
          Range:
            'bytes=' +
            tileMeta.offset +
            '-' +
            (tileMeta.offset + tileMeta.length - 1),
        },
      })
      const data = await response.blob()
      tile.getImage().src = URL.createObjectURL(data)
    } catch (e) {
      tile.setState(TileState.ERROR)
    }
  }
}

I think this should serve as a good starting point for a fully-featured plugin. However, there is still the "workaround" using DataURLs in there, which i don't particularly like. This won't be necessary once the commit i linked above is merged into openlayers, which allows use to directly create tiles from the binary data!

Apr 26 '21 20:04 fl0cke

@fl0cke thanks, I got this running! What do you think is the right plugin structure? I don't use OpenLayers much but I had the impression from https://openlayers.org/en/latest/doc/tutorials/bundle.html that ES6 modules are strongly preferred by OL users, so here's some ideas I have:

I would prefer that pmtiles.js be a plain JS library and not depend on OL, and pmtiles.js should be usable by script includes (as IIFE instead of es6 module) for basic use cases
for OL support we can have a separate ES6 module that depends on the pmtiles.mjs module as well as OL, either in this repo or a totally separate repo (openlayers-pmtiles)
I would prefer that the boilerplate for instantiating a TileLayer etc be non-specific to images, that way it can also be provided by a https://github.com/protomaps/protomaps.js -rendered Canvas element instead of a image data url (any ideas on how to make your example work with Canvas?)

May 08 '21 07:05 bdon

An example now exists of vector tiles loaded in OpenLayers using the bare PMTiles class:

live demo: https://protomaps.github.io/PMTiles/examples/openlayers.html

source code: js/examples/openlayers.html

This doesn't feel super fluid right now, I suspect it may have to do with request cancellation when moving more than one zoom level. Anyways, to raise this into a first-class Source for OL would require abstracting away the hardcoded details in that code snippet.

Oct 28 '22 02:10 bdon

some small changes in https://github.com/protomaps/PMTiles/commit/674e92624dfa3cd4aded79535b0b20d9ae9d077a seem to improve the experience by setting TileState

Oct 28 '22 02:10 bdon

Example ad-hoc OL integration in OL repo: https://github.com/openlayers/openlayers/pull/14256

Nov 04 '22 11:11 bdon

These recent changes in OpenLayers (in v7.2.2, I believe) really improved the bug of map tiles stopping loading when zooming in and out. It's still there, but now it's harder to break.

What I noticed, though, that protomaps-themes styles still tend to break loading. I'm testing OpenLayers on two PMTiles maps:

Custom file generated from raw osm.pbf files with OpenMapTiles styles (one full country, ~2GB file)
Map downloaded through https://app.protomaps.com/downloads/small_map with protomaps-themes style adapted to OpenLayers with ol-mapbox-style (10MB file of small city)

My custom 2GB map works almost perfectly, zooming in and out between separate corners of the map as fast as possible, loads tiles pretty quick.

Small map glitches all the time, refusing to load tiles after little zoom changes (let's say zooming from 5 to 10 is enough to break it)

Feb 20 '23 20:02 patrykkalinowski

@patrykkalinowski can you provide a video of the issue? That seems odd that the style is affecting the behavior that way, if all other parts of the code are the same.

Feb 21 '23 02:02 bdon

Update: I think this might not be related to styles, my bad. I removed styles from all these 3 examples and the same bug occured with raw vector data. My map works well, while Protomaps keep glitching.

Sure thing @bdon, here are the recordings:

You can see that after few zooms on protomaps (2 and 3), tiles stop loading.

As for the code, I generated style file using these instructions: https://protomaps.com/docs/frontends/maplibre#vector-basemaps

import layers from 'protomaps-themes-base';

export function glStyle(pmtilesUrl = "http://example.com", colors = "light") {
  let glStyle = {
    version:8,
    glyphs:'https://cdn.protomaps.com/fonts/pbf/{fontstack}/{range}.pbf',
    sources: {
        "protomaps": {
            type: "vector",
            url: pmtilesUrl,
            attribution: '<a href="https://protomaps.com">Protomaps</a> © <a href="https://openstreetmap.org">OpenStreetMap</a>'
        }
    },
    layers: layers("protomaps", colors)
  }

  return glStyle
}

And applying the style to OL map:

MapboxStyle.stylefunction($map.layers.basemap, glStyle(), "protomaps")

And this is how I load openmaptiles style to my custom map:

fetch('/styles/topo-v2.json')
        .then((response) => response.json())
        .then((glStyle) => {
          MapboxStyle.stylefunction($map.layers.basemap, glStyle, "openmaptiles")
        })

Hope this helps

Feb 21 '23 21:02 patrykkalinowski

@patrykkalinowski I want to watch the videos, but they seem to be private or offline! can you re-upload them?

Feb 24 '23 07:02 bdon

Sorry, they probably expired. Uploaded again on youtube:

Feb 24 '23 23:02 patrykkalinowski

@patrykkalinowski interesting, I'll have to take a closer look...

Can you try adding maxzoom : 10 to the protomaps object in the GL style? would be also helpful to record any console error messages. Possible requests for missing tiles are causing errors and disrupting the tile rendering queue somehow.

Feb 28 '23 01:02 bdon

There are no error messages or anything logged in console. It just stops requesting new tiles (no more requests are visible in Network tab).

What helped was to limit map view to extent. If I stay within map boundaries, it seems like everything is much more reliable. Quickly zooming in now breakes tile loading only on high zoom levels and when tiles are fetched from online source.

What fixes the issue completely is combination of view limited to extent and offline map source. When I load PMTiles file as blob from IndexedDB, they never bug out. Seems like OL has issues when previous tiles were not loaded on time before next requests appear?

Anyways, I'll try limiting maxzoom and report on the results.

Mar 07 '23 21:03 patrykkalinowski

Another thing I've noticed, that protomaps tile loading bugs out also when switching map layers. Full refresh of the page (and loading OL from scratch) fixes the issue.

This is my code for loading (and switching) maps:

ol_map.removeLayer($map.layers.basemap)

newURL = URL.createObjectURL(map.blob)

// create new basemap source and layer
$map.sources.basemap = new PMTiles.OLPMTilesSource(newURL)
$map.layers.basemap = new VectorTileLayer({
  source: new VectorTileSource({
    format: new MVT(), // this could come from the PMTiles header (async)
    url: $map.sources.basemap.url(),
    tileLoadFunction: $map.sources.basemap.vectorTileLoadFunction
  }),
  declutter: true
})

$map.ol_map.addLayer($map.layers.basemap)

I didn't find the root cause yet.

Mar 08 '23 10:03 patrykkalinowski

To confirm: you are using https://github.com/protomaps/PMTiles/blob/master/js/examples/openlayers.html exactly for OLPMTilesSource?

I'm skeptical that the Style itself has anything to do with this, since it should be simpler than most other styles - unless it has to do with Fonts

Mar 09 '23 07:03 bdon

Yes, I'm using exactly this code.

You may have missed a paragraph in one of previous posts:

I think this might not be related to styles, my bad. I removed styles from all these 3 examples and the same bug occured with raw vector data. My map works well, while Protomaps keep glitching.

Behaviour is the same without styles, here is an example of zoomed in map with raw vector data, no style:

Tiles simply stop loading without any errors. When I reload the website completely, it works.

I tried to completely rebuild the Map object (basically recreating the process of initial load) instead of just changing vector layer but the issue persisted. Currently I'm out of ideas, but I'll let you know if I figure something out.

Mar 09 '23 20:03 patrykkalinowski

I'm not confident in way OLPMTilesSource is implemented as-is. We should make the integration look like this in the OL repo: https://github.com/openlayers/openlayers/blob/d7c9c954d62832d9f5b9f8ec3383ab99fba685c9/examples/pmtiles-elevation.js

See the discussion on https://github.com/openlayers/openlayers/pull/14256 for context. I'm not following OL closely, but it does seem like we want to use the newer interface for DataTile for vector tiles.

Mar 12 '23 05:03 bdon

Hi @patrykkalinowski - can you try the change by @weskamm in https://github.com/protomaps/PMTiles/pull/142 to see if that resolves your tile loading issue?

Mar 23 '23 01:03 bdon

hey @bdon, it actually made things worse.

With this change, map disappears completely on zoom levels which were not loaded correctly.

Mar 23 '23 16:03 patrykkalinowski

@patrykkalinowski based on my experiments I'm almost certain that you need to specify a maxZoom in the constructor of ol.source.VectorTile here:

$map.layers.basemap = new VectorTileLayer({
  source: new VectorTileSource({
    format: new MVT(), // this could come from the PMTiles header (async)
    url: $map.sources.basemap.url(),
    tileLoadFunction: $map.sources.basemap.vectorTileLoadFunction
  }),
  declutter: true
})

the maxZoom needs to match exactly the max tile zoom level available in your PMTiles archive. (This should really come from the header but the example creates the map/tilesource object synchronously)

Mar 26 '23 08:03 bdon

@bdon thanks, that did the trick! with maxZoom property no more missing tiles, everything works like expected :)

Mar 26 '23 14:03 patrykkalinowski

Thanks to @tschaub I've updated the OpenLayers snippet integrations:

https://github.com/protomaps/PMTiles/blob/main/js/examples/openlayers.html https://github.com/protomaps/PMTiles/blob/main/js/examples/openlayers_raster.html

These now asynchronously load the PMTiles metadata to automatically determine minZoom and maxZoom. the raster tile example also uses the WebGL layer.

Apr 26 '23 14:04 bdon

Hi all, I've published a very basic ol-pmtiles package with example usage here:

https://github.com/protomaps/PMTiles/tree/main/openlayers

Current TODOs:

Explicitly pass more source creation options through the constructor.
The current packaging on NPM is source distribution only and assumes you are using a build step and bundler with OpenLayers as recommended by the OL documentation. There is no existing provision for single script tag includes (IIFE). This is difficult to accomplish because we want to bundle the pmtiles dependency and not the ol dependency (it will bring in 200KB+ uncompressed of code because of OL's extensibility via inheritance, and that will be code for a specific point version of OL)

Comment here if you're able to find this useful.

May 05 '23 09:05 bdon

bundle the pmtiles dependency and not the ol dependency

@bdon I'm not sure if every bundler supports it, but at least in Rollup I use a combination of output.globals + external config which designates selected dependencies as external or peer, so that they are not bundled together, but expected to be available in runtime.

May 05 '23 09:05 zakjan

@zakjan do you have an example npm package for OL bundled with rollup? The rest of the tools in this repo use esbuild but the OL package is intentionally separate so we can use rollup if necessary.

May 05 '23 10:05 bdon

It's not OL but Leaflet, but the principle is the same. https://github.com/zakjan/leaflet-lasso/blob/master/rollup.config.js

May 05 '23 10:05 zakjan

@zakjan thanks! I think that illustrates the issue however - Leaflet happens to work with this because references to the L are the same in both bundled and standalone code. For OL we need to import specific classes like this but we would need to refer to ol.source.VectorTile in a standalone script includes. So I don't see any easy way to get both for OL.

May 08 '23 02:05 bdon

Would it work if you declare ol/source/VectorTile as external + alias ol/source/VectorTile to a global ol.source.VectorTile?

May 10 '23 11:05 zakjan

@zakjan I accomplished this by making a copy of the source code into a new script_includes.js with manually edited imports - a hacky solution, but this is < 100 LOC total.

https://github.com/protomaps/PMTiles/tree/main/openlayers and READMEs have been updated with samples on how to use the new NPM module. Closing this issue - if anyone encounters bugs or missing functionality please open a new issue specific to that.

May 10 '23 12:05 bdon