Style spec: Pull more than one data source into a layer

[kkaefer]

Style layers can currently only specify a single data layer. With data driven styling, we're looking at rendering the same data with fewer style layers. I propose adding support for pulling multiple data sources into one style layer. In a data source, the split between putting features into separate data layers vs. putting them in the same data layer and using tags to filter/separate them is somewhat arbitrary. Adding support for multiple data layers also means that they could stem from different sources, e.g. a regular tileset and a GeoJSON file, or runtime annotations.

In terms of style spec change, we could make the source field an array that allows specifying multiple data source/layer combinations:

{
  "id": "landcover",
  "type": "fill",
  "source": [
    ["streets", "landcover"],
    ["geojson", "glaciers"]
  ],
  ...
}

Alternatively, we could use a different attribute like data and move source, source-layer, and filter to that:

{
  "id": "landcover",
  "type": "fill",
  "data": [
    {
      "source": "composite",
      "source-layer": "landcover",
      "filter": [
        "all",
        ["in", "class", "crop", "grass", "scrub", "wood", "snow"]
      ]
    },
    {
      "source": "glaciers.geojson"
    }
  ],
  ...
}

[jfirebaugh]

Previously:

• https://github.com/mapbox/mapbox-gl-style-spec/issues/224
• https://github.com/mapbox/mapbox-gl-style-spec/issues/231

[mourner]

Combining several sources in one layer might be a big challenge because of the need to synchronize data between workers (e.g. if you have VT + GeoJSON). However, implementing support for pulling from multiple source layers in one source could be much easier. We could e.g. optionally move the source-layer property inside the filter syntax for that:

{
  "source": "composite",
  "filter": [
    "any",
    ["all", ["==", "$layer", "landcover"], ["in", "class", "crop", "grass", "scrub", "wood", "snow"]],
    ["all", ["==", "$layer", "hillshade"], ["in", "foo", "bar"]]
  ]
}

[anderslarssonvbg]

Is this something that might come in a not too far future?

[asheemmamoowala]

Brought forward from #9317 by @sansumbrella

Motivation

I want to use heatmaps and cluster visualizations to display historic data grouped in different time intervals. I already have my data stored in a series of 1-day groups. At runtime, I want to be able to choose what timeframe to visualize: any single day, or any span of seven or thirty days. Because of my visual design choices, the source data needs to be grouped together for display on a single layer.

Design Alternatives

Store all possible data in a single source and filter the data in an expression. This results in always downloading the most possible data and puts a potentially large computing cost on the client device.

Manage source aggregation earlier in the pipeline and change the layer source to the desired aggregation at runtime. I opened a separate ticket for enabling this kind of aggregation when creating tilesets.

Ideally, source aggregation would be supported at multiple points in the application lifecycle to allow for different use cases. Currently application developers must develop their own aggregation scheme and provide a single source when they interface with Mapbox tools.

Design

In the layer style specification, add a sources key that accepts an array of sources. If an array of sources is provided, a singular source and source-layer should not be provided.

Mock-Up

Library developers will support a new variant of the Layer type that contains multiple sources.

interface Source {
  id: string; // Name of a source description to be used in this layer
  layer?: string; // Layer to use from a vector tile source
}

interface LayerBase {
  // …
  // existing style spec for layers omitting source, source-layer, and sources
}

interface LayerWithSingleSource extends LayerBase {
  source: string;
  "source-layer"?: string;
}

interface LayerWithMultipleSources extends LayerBase {
  sources: Source[];
}

type Layer = LayerWithSingleSource | LayerWithMultipleSources;

Application developers can add an array of sources instead of a single source+source-layer combo.

{
  type: "heatmap"
  sources: [ { id: "vehicles-2020-02-16", layer: "vehicles" }, { id: "vehicles-2020-02-17", layer: "vehicles" }, { id: "vehicles-2020-02-18", layer: "vehicles" } ]
}

Concepts

New concept: you can have multiple sources for a single visual layer.

The sources should have compatible feature types for the style layer using them (e.g. all lines or polygons for a line layer).

Implementation

In a simplest implementation, an AggregateSource acts as a facade to a collection of underlying sources. When data is requested from the source, it passes the request to all sources in the collection, aggregates their responses into a single response object, and provides that aggregate response to the code requesting data. Styles and other consumers of sources should not need to know anything about the source or how it works.

Pseudocode ignoring asynchrony and other library details:

class AggregateSource {
  sources: Source[];

  loadTile(tile: Tile): Data[] {
    return this.sources.reduce((aggregate, source) => {
      return aggregate.concat(source.loadTile(tile));
    }, []);
  }
}

[daniela-green]

Hey, any news on that? I want to load 2 different sources using the same layer.

Thanks!

[mconradiesa]

I would love this feature. I've got 238 countries as separate sources and have to specify a separate set of styles for each country. I could merge each feature type into one using ogr2ogr by that'll take months.

[gregorwebdev]

Any update on this? I generate pbf tiles, where 90% of the data is stable, and 10% is changing more often. If a layer could use more sources, than I could cache the stable data tiles, and regenerate only the 10% data every time. I have already more than enough layer, so i wouldn't seperate the data into different layers. I tought this is the way to do this, but not:
"tiles":` [ "http://a.example.com/tiles/{z}/{x}/{y}.pbf", "http://b.example.com/tiles/{z}/{x}/{y}.pbf" ], As I understood this is for tile serving optimalisation. Or any workaround for this? I would develop the libary if somebody helps a bit.