feat(plugin): add plugin-chart-cartodiagram

[jansule]

SUMMARY

This adds a new plugin plugin-chart-cartodiagram that allows creating cartodiagrams, i.e. charts on a map. See also https://github.com/apache/superset/discussions/21758.

How does it work?

In order to support as many charts as possible, we created some kind of a meta plugin that takes configurations of any existing chart plugin supported by superset and places it on a map. To do so, we make all charts with a common data source selectable, and inject a group-by property based on a provided geometry column. Then, we use the buildQuery() and transformProps() functions of each chart in combination with the respective chart component (via the ChartComponentRegistry) to render them on a map. Through this, all configuration options of any chart are supported.

Additional configuration options for cartodiagrams are:

• Map extent: Either fit the view to the extent of the chart data (every item should be visible), or defined a custom extent by moving the map to the desired location.
• Background layers: Add an arbitrary number of background layers to the map. Currently the formats WMS (raster), XYZ (raster) and WFS (vector) in multiple versions are supported. Vector layers can be styled in the explore view.
• Background color of chart: The background color and opacity of the chart can be configured in order to improve the readability of a chart depending on the configured background layers.
• Corner radius of chart: Set the corner radius of the wrapping <div> element of a chart.
• Zoom based chart size: Set the chart size for each zoom level in order to avoid cluttering. Predefined functions to compute all values are provided (fixed, linear, exponential).

Default layers can be configured in MainPreset.js which will be shown by default for every configured cartodiagram panel. Users are able to edit/remove these layers for a single panel in the explore view. Default layers currently include a WMS showing OpenStreetMap data in grayscale.

Disclaimer: The service providing the OpenStreetMap data is a demo service hosted by terrestris GmbH & Co. KG, which is my employer. On a higher zoomlevel, it shows a watermark for our paid service. The demo server is not recommended to be used in production and we recommend replacing this service. It was just added for others to have a default background map while getting familiar with the plugin and probably should be replaced in the future.

Technical Details/Requirements

Additional libraries:

• OpenLayers as mapping library
• GeoStyler for styling of vector data (disclaimer: I am one of the maintainers of GeoStyler)

Both libraries are part of the Open Source Geospatial Foundation, so the chances of future development of those libraries are quite certain.

---

In order to make use of above libraries, we had to update some dependencies (e.g. typescript, see also https://github.com/apache/superset/pull/24272), and we also had to make use of npm dedupe in order to resolve version conflicts. Unfortunately, this resulted in a nearly complete rewrite of the package-lock.json.

Due to some version upgrades, we also had to apply some linting fixes, which results in more changed files than actually directly related to this PR. Sorry for that.

---

The <SelectControl> had to be changed a little to be able to add a customized list of options. We use it to display all charts with a common dataset and also check, if a previously selected chart was updated after the selection.

Limitations

There are some limitations of the plugin, which probably should be addressed in the future:

• We currently only support geometries in the coordinate reference system (CRS) ~EPSG:3857 (WGS 84 / Pseudo-Mercator)~ EPSG:4326 (WGS 84). Additional CRSs might be added in the future.
• For now, we expect the geometry column to return a GeoJSON point geometry for placing the charts on the map. Additional geometries (e.g. lines and polygons), as well as additional formats (WellKnownText, lat/lon, etc.) might be added in the future.
• Rendering performance can be improved and varies strongly depending on the used chart type.
• ~Currently, filters applied to a dashboard are not reflected in the cartodiagram plugin. We would be grateful, if someone can hint us to the right direction for supporting this.~ Dashboard filters are now supported.

---

Since the background layers point to external services, the CSP have to be adjusted. Additional information can be found in the README of the plugin.

How can this be improved?

This PR is just the first step into the direction for additional geospatial support. So there are many things that can be added and improved. Besides the points mentioned above, following points might also be useful:

• Adding a FeatureFlag to chart plugins that should be allowed to place on a map is probably a nice feature that was also already discussed with @villebro. To do so, the filter options of the REST API need to be extended.
• Improved caching of the charts in the client, in order to reduce the number of rerenderings of a chart.
• Adding the geometries of the actual dataset as a background layer and allow styling it.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Example dashboard with some cartodiagrams:

Explore view of cartodiagram plugin (configuration):

Explore view of cartodiagram plugin (customization):

Explore view of cartodiagram plugin (layer configuration):

Explore view of cartodiagram plugin (layer styling):

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

• [x] Has associated issue: https://github.com/apache/superset/discussions/21758
• [ ] Required feature flags:
• [ ] Changes UI
• [ ] Includes DB Migration (follow approval process in SIP-59)
  • [ ] Migration is atomic, supports rollback & is backwards-compatible
  • [ ] Confirm DB migration upgrade and downgrade tested
  • [ ] Runtime estimates and downtime expectations provided
• [x] Introduces new feature or API
• [ ] Removes existing feature or API

[jansule]

I'm not sure why the linting in the Frontend/frontend-build pipeline fails here. On my local setup, I do not get any errors (using node v16.20.2).

[justinpark]

@jansule Thanks for contributing a new chart plugin. This PR is quite large and includes TypeScript changes. Could you please detach all changes not related to the newly introduced chart plugin?

[jansule]

@jansule Thanks for contributing a new chart plugin. This PR is quite large and includes TypeScript changes. Could you please detach all changes not related to the newly introduced chart plugin?

Thanks for the first review @justinpark. I created a PR with only the TypeScript changes a while ago. See https://github.com/apache/superset/pull/24272. I guess it just got lost in the amount of PRs you are receiving.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (2cd7135) 67.17% compared to head (a544528) 69.33%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #25869      +/-   ##
==========================================
+ Coverage   67.17%   69.33%   +2.16%     
==========================================
  Files        1899     1895       -4     
  Lines       74354    74279      -75     
  Branches     8266     8248      -18     
==========================================
+ Hits        49945    51501    +1556     
+ Misses      22360    20735    -1625     
+ Partials     2049     2043       -6     

Flag	Coverage Δ
hive	53.80% <ø> (?)
postgres	?
presto	53.75% <ø> (?)
python	82.71% <ø> (+4.52%)	:arrow_up:
sqlite	77.68% <ø> (ø)
unit	56.42% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jansule]

@villebro I finally managed to update the PR so that the pipelines succeed. The failing Dependency Review pipeline fails due to a missing license entry in one of our transitive dependencies (BSD-3 Clause). I added a PR there a while ago that fixes the problem.

So could you take another look at this PR?

[lcalisto]

Any update on this PR? This would be a very cool feature for Superset.

[pesekon2]

I would also like to have this PR merged (mapbox is a joke). All my support to @jansule for the patience with updating and conflict solving, etc.

I've tried to test it locally, although not being eligible as a possible reviewer. It seems to work. I am not able to load any WMS as the background but this could possibly be due to some of the proxy/reverseproxy black magic here.

[pesekon2]

I would also like to have this PR merged (mapbox is a joke). All my support to @jansule for the patience with updating and conflict solving, etc.

I've tried to test it locally, although not being eligible as a possible reviewer. It seems to work. I am not able to load any WMS as the background but this could possibly be due to some of the proxy/reverseproxy black magic here.

Not a proxy magic. Made it work locally with the default terreatris WMS but not being able to change the WMS to anything else. See the example in the screenshot (no WMS visible, no error message in logs; the service URL is https://eagri.cz/public/app/wms/public_zp.fcgi):

[rusackas]

Looks like css-font-parser is making into package-lock.json and has an incompatible license. Any way to get rid of that?

[jansule]

Thanks for the feedback! I will try to get some time to resolve the merge conflicts soon.

Looks like css-font-parser is making into package-lock.json and has an incompatible license. Any way to get rid of that?

@rusackas There is an invalid spdx identifier in css-font-parser, but the license itself is compatible (BSD-3-Clause). I created a PR a while ago to solve the issue https://github.com/bramstein/css-font-parser/pull/11 but did not get any feedback.

I am not able to load any WMS as the background but this could possibly be due to some of the proxy/reverseproxy black magic here.

@lcalisto So far I had no problems with loading other WMS. You might want to ensure that your Talisman settings are correct. I will double check as soon as I get time to work on this PR again.

[pesekon2]

I am not able to load any WMS as the background but this could possibly be due to some of the proxy/reverseproxy black magic here.

@lcalisto So far I had no problems with loading other WMS. You might want to ensure that your Talisman settings are correct. I will double check as soon as I get time to work on this PR again.

Solved it today (I guess it was on me and not on @lcalisto ). It was indeed a problem with Talisman. I didn't make it work with the Talisman so simply turning it off helped but this could very probably be connected to some security issues at the server or some obscure thing about the WMS in use.

Therefore: Big thumbs up. Tested and working for me. Thanks very much and I hope it will get merged one day.

[rusackas]

@rusackas There is an invalid spdx identifier in css-font-parser, but the license itself is compatible (BSD-3-Clause). I created a PR a while ago to solve the issue https://github.com/bramstein/css-font-parser/pull/11 but did not get any feedback.

Ahh, ok, if it's all good, you can get the Dependency Review CI check to pass by adding the package to allow-dependencies-licenses in the .github/workflows/dependency-review.yml file

[lcalisto]

I just saw that @jansule PR https://github.com/bramstein/css-font-parser/pull/11 has been merged. Perhaps that also helps. 🙂

[jansule]

I just saw that @jansule PR bramstein/css-font-parser#11 has been merged. Perhaps that also helps. 🙂

That's great. I justed pinged the owner to create a new release. Hopefully this will happen soon. Otherwise I will add the dependeny to the allow-dependencies-licenses as @rusackas mentioned.

Solved it today (I guess it was on me and not on @lcalisto ). It was indeed a problem with Talisman. I didn't make it work with the Talisman so simply turning it off helped but this could very probably be connected to some security issues at the server or some obscure thing about the WMS in use.

Happy to hear that it's working for you now!

[jansule]

@rusackas @villebro I just rebased the PR. Could you take another look?

[jansule]

Thanks for the review @villebro! I applied your suggestions. It would be great if you could have another look at it

[jansule]

The frontend-build fails due to @types/geojson not being listed as a dependency of plugins/legacy-preset-chart-deckgl. Is this something that should be addressed within this PR?

[villebro]

The frontend-build fails due to @types/geojson not being listed as a dependency of plugins/legacy-preset-chart-deckgl. Is this something that should be addressed within this PR?

@jansule hmm, this is curious - let's satisfy the linter and see if it all makes sense when CI is green.

[jansule]

@jansule hmm, this is curious - let's satisfy the linter and see if it all makes sense when CI is green.

Done. Looks like the dependency was actually missing. I'm wondering why this check did not fail before.

[rusackas]

I'm excited we can likely get this into 5.0!

[kgabryje]

Hi @jansule! I noticed that this PR has added a huge chunk to the bundle (2.14MB parsed, visible in the top left corner in the first screenshot). For some reason it contains all antd icons (700kB) and pages/Chart module (500kB), which makes this chunk the largest one in the bundle. For comparison, second screenshot shows the bundle analysis after reverting this PR. It probably won't affect the loading times of the dashboards which don't contain the Cartodiagram chart, but we might want to solve this issue for the sake of loading performance of dashboards that use this plugin (especially for users with slower Internet)

[jansule]

Thanks for the analysis @kgabryje. I will take a closer look at that

[kgabryje]

Awesome, thank you!