Magda Integration with NationalMap V8

[t83714]

Magda Integration with NationalMap V8

This is now waiting on:

• [ ] Nationalmap v8 release
• [ ] PR & release on DGA magda

---

Original ticket:

There are a few things we might want to check in order to make sure Magda still works well with NationalMap V8:

• [x] Magda currently sends Catalogue config to National Map via postMessage approach and we are sending a magda-item type catalog item to nationalMap see here. We need to check:
  • Does the postMessage approach still work
  • Does the current config data sent over works with v8
    • As v8 use MagdaReference.ts to handle magda-item, we want to make sure its functionality is on a par with previous MagdaCatalogItem.js
  • I think it's a high priority item to check
• [ ] As Magda now can host data files in its internal storage and some of data files might be protected, to make Magda Catalogue item still works with those internal data files, Terria’s ajax requests need to include cookie headers (assuming users already logged in to Magda). If we use xhr, we need to set withCredentials to true. If we use fetch api, we need to set credentials to include. Magda Catalogue Item s are previously handled by MagdaCatalogItem.js but in v8, it's MagdaReference.ts.
  • I think it's a low priority item to solve if it's too hard as I can hide the Open in National Map Button for internal data files
• [ ] As Magda now can host data files in its internal storage, I have upgraded the MagdaCatalogItem.js that is used by Magda's Map Preview Module. We probably want to migrate the similar changes to MagdaReference.ts.
  • I think it's a low priority item to solve and only need to solve if we solve the item above

[AnaBelgun]

@mwu2018 The integration needs to be checked both ways:

• data (on NationalMap) should be visualised on data.gov.au if geospatial
• data in data.gov.au should be harvested in NationalMap; currently there might be discrepancies between the whole geospatial data catalogue in data.gov.au and what is available in NationalMap v7 - TBC

[t83714]

Test site for v8: https://nationalmap.dev.saas.terria.io/

[t83714]

Source Code Open National Map Button: https://github.com/magda-io/magda/blob/2dd76a04bf3d6ff7261c14531098e8ac9e32ca21/magda-web-client/src/Components/Common/DataPreviewMapOpenInNationalMapButton.js#L65

[mwu2018]

@mwu2018 The integration needs to be checked both ways:

• data (on NationalMap) should be visualised on data.gov.au if geospatial
• data in data.gov.au should be harvested in NationalMap; currently there might be discrepancies between the whole geospatial data catalogue in data.gov.au and what is available in NationalMap v7 - TBC

On the first point, @t83714 has done an initial test that shows v8 national map test site does not support postMessage approach at the moment. @soyarsauce advises that it is because postMessage flow has not been set up in v8 yet and needs to be fixed.

@soyarsauce advises that the magda-item should be changed to magda in the following code.

const catConfig = {
            initSources: [
                {
                    catalog: [
                        {
                            name: this.props.distribution.title,
                            type: "magda-item",
                            distributionId: this.props.distribution.identifier,
                            url: config.baseExternalUrl,
                            isEnabled: true,
                            zoomOnEnable: true
                        }
                    ]
                }
            ]
        };

On the second point, because the national map queries ckan with a set of filters, it is quite normal that the number of ckan items will be different from that in data.gov.au.

See below:

Jacky Jiang  2 hours ago
If it’s that one, it pull directly from DGA ckan using this query:
fq: +(res_format:(geojson OR GeoJSON OR kml OR KML OR kmz OR KMZ OR wms OR WMS OR CSV-GEO-AU OR csv-geo-au OR aus-geo-csv OR "Esri REST")) -(organization:alpine-shire-council OR organization:brimbank-city-council OR organization:brisbane-city-council OR organization:bundaberg-regional-council OR organization:cairns-regional-council OR organization:cardinia-shire-council OR organization:central-goldfields-shire-council OR organization:city-of-ballarat OR organization:city-of-boroondara OR organization:city-of-casey OR organization:city-of-darwin OR organization:city-of-gold-coast OR organization:city-of-greater-bendigo OR organization:city-of-greater-dandenong OR organization:city-of-greater-geelong OR organization:city-of-hobart OR organization:city-of-launceston OR organization:cityofmelbourne OR organization:city-of-monash OR organization:city-of-palmerston OR organization:city-of-port-phillip OR organization:city-of-stonnington OR organization:city-of-whittlesea OR organization:city-of-yarra OR organization:colac-otway-shire OR organization:corangamite-shire-council OR organization:frankston-city-council OR organization:fraser-coast-regional-council OR organization:glen-eira-city-council OR organization:glenelg-shire-council OR organization:gcc OR organization:golden-plains-shire-council OR organization:greater-shepparton-city-council OR organization:hindmarsh-shire-council OR organization:hobsons-bay-city-council OR organization:horsham-rural-city-council OR organization:city-of-ipswich OR organization:lake-macquarie-city-council OR organization:latrobe-city-council OR organization:loddon-shire-council OR organization:logan-city-council OR organization:macedon-ranges-shire-council OR organization:manningham-city-council OR organization:mansfield-shire-council OR organization:maribyrnong-city-council OR organization:melton-city-council OR organization:mildura-rural-city-council OR organization:www-mitchellshirecouncil-vic-gov-au OR organization:moorabool-shire-council OR organization:moreton-bay-regional-council OR organization:mosmanmunicipalcouncil OR organization:nillumbik-shire-council OR organization:noosa-shire-council OR organization:northern-grampians-shire-council OR organization:southern-grampians-shire-council OR organization:strathbogie-shire-council OR organization:surf-coast-shire-council OR organization:swan-hill-rural-city-council OR organization:toowoomba-regional-council OR organization:townsville-city-council OR organization:wangaratta-rural-city-council OR organization:warrnambool-city-council OR organization:west-wimmera-shire-council OR organization:wodonga-city-council OR organization:wyndham-city-council OR organization:yarra-ranges-council OR organization:yarra-valley-water OR organization:yarriambiack-shire-council) -(organization:australian-institute-of-marine-science OR organization:geoscienceaustralia OR organization:csiro-oceans-and-atmosphere-information-and-data-centre OR organization:csiro-marine-and-amospheric-research OR organization:state-of-the-environment OR organization:new-south-wales-datasets OR organization:south-australian-governments)

[t83714]

This is the relevant issue that may impact item one (open national map button on DGA): https://github.com/TerriaJS/terriajs/issues/4880

[soyarsauce]

Related: https://github.com/TerriaJS/terriajs/issues/4904

[mwu2018]

This ticket at least depends on the following tickets:

• [ ] https://github.com/magda-io/magda/issues/3019. (This PR is ready.)

• [x] https://github.com/TerriaJS/terriajs/issues/4906

• [x] https://github.com/TerriaJS/TerriaMap/issues/494

• [x] Sync terrace (a fork of TerriaMap) to upstream TerriaMap (https://github.com/TerriaJS/terrace/issues/151)

• [ ] Release of v8 NationalMap

The following tickets are not critical but desirable.

• [ ] https://github.com/TerriaJS/terriajs/issues/4904 (This PR is ready.)

Before the NationalMap is upgraded to v8 terriajs, this site can be used for testing.

~With the above tickets resolved, the magda record might not automatically show up on the map. See Technical note for developers for further development. A related issue is created here~

About dataset discrepancies between DGA and NationalMap @AnaBelgun

The NationalMap queries a subset of datasets from DGA, only requesting datasets with specific formats and excluding datasets from some organisations. Of the specified formats, there are about 1168 datasets in DGA:

https://www.data.gov.au/api/3/action/package_search?rows=100000&sort=metadata_created%20asc&fq=%2B(res_format%3A(geojson%20OR%20GeoJSON%20OR%20kml%20OR%20KML%20OR%20kmz%20OR%20KMZ%20OR%20wms%20OR%20WMS%20OR%20CSV-GEO-AU%20OR%20csv-geo-au%20OR%20aus-geo-csv%20OR%20%22Esri%20REST%22))

Because of the exclusion of some organisations, the NationalMap only obtains about 242 datasets:

https://www.data.gov.au/api/3/action/package_search?rows=100000&sort=metadata_created%20asc&fq=%2B(res_format%3A(geojson%20OR%20GeoJSON%20OR%20kml%20OR%20KML%20OR%20kmz%20OR%20KMZ%20OR%20wms%20OR%20WMS%20OR%20CSV-GEO-AU%20OR%20csv-geo-au%20OR%20aus-geo-csv%20OR%20%22Esri%20REST%22))%20-(organization%3Aalpine-shire-council%20OR%20organization%3Abrimbank-city-council%20OR%20organization%3Abrisbane-city-council%20OR%20organization%3Abundaberg-regional-council%20OR%20organization%3Acairns-regional-council%20OR%20organization%3Acardinia-shire-council%20OR%20organization%3Acentral-goldfields-shire-council%20OR%20organization%3Acity-of-ballarat%20OR%20organization%3Acity-of-boroondara%20OR%20organization%3Acity-of-casey%20OR%20organization%3Acity-of-darwin%20OR%20organization%3Acity-of-gold-coast%20OR%20organization%3Acity-of-greater-bendigo%20OR%20organization%3Acity-of-greater-dandenong%20OR%20organization%3Acity-of-greater-geelong%20OR%20organization%3Acity-of-hobart%20OR%20organization%3Acity-of-launceston%20OR%20organization%3Acityofmelbourne%20OR%20organization%3Acity-of-monash%20OR%20organization%3Acity-of-palmerston%20OR%20organization%3Acity-of-port-phillip%20OR%20organization%3Acity-of-stonnington%20OR%20organization%3Acity-of-whittlesea%20OR%20organization%3Acity-of-yarra%20OR%20organization%3Acolac-otway-shire%20OR%20organization%3Acorangamite-shire-council%20OR%20organization%3Afrankston-city-council%20OR%20organization%3Afraser-coast-regional-council%20OR%20organization%3Aglen-eira-city-council%20OR%20organization%3Aglenelg-shire-council%20OR%20organization%3Agcc%20OR%20organization%3Agolden-plains-shire-council%20OR%20organization%3Agreater-shepparton-city-council%20OR%20organization%3Ahindmarsh-shire-council%20OR%20organization%3Ahobsons-bay-city-council%20OR%20organization%3Ahorsham-rural-city-council%20OR%20organization%3Acity-of-ipswich%20OR%20organization%3Alake-macquarie-city-council%20OR%20organization%3Alatrobe-city-council%20OR%20organization%3Aloddon-shire-council%20OR%20organization%3Alogan-city-council%20OR%20organization%3Amacedon-ranges-shire-council%20OR%20organization%3Amanningham-city-council%20OR%20organization%3Amansfield-shire-council%20OR%20organization%3Amaribyrnong-city-council%20OR%20organization%3Amelton-city-council%20OR%20organization%3Amildura-rural-city-council%20OR%20organization%3Awww-mitchellshirecouncil-vic-gov-au%20OR%20organization%3Amoorabool-shire-council%20OR%20organization%3Amoreton-bay-regional-council%20OR%20organization%3Amosmanmunicipalcouncil%20OR%20organization%3Anillumbik-shire-council%20OR%20organization%3Anoosa-shire-council%20OR%20organization%3Anorthern-grampians-shire-council%20OR%20organization%3Asouthern-grampians-shire-council%20OR%20organization%3Astrathbogie-shire-council%20OR%20organization%3Asurf-coast-shire-council%20OR%20organization%3Aswan-hill-rural-city-council%20OR%20organization%3Atoowoomba-regional-council%20OR%20organization%3Atownsville-city-council%20OR%20organization%3Awangaratta-rural-city-council%20OR%20organization%3Awarrnambool-city-council%20OR%20organization%3Awest-wimmera-shire-council%20OR%20organization%3Awodonga-city-council%20OR%20organization%3Awyndham-city-council%20OR%20organization%3Ayarra-ranges-council%20OR%20organization%3Ayarra-valley-water%20OR%20organization%3Ayarriambiack-shire-council)%20-(organization%3Aaustralian-institute-of-marine-science%20OR%20organization%3Ageoscienceaustralia%20OR%20organization%3Acsiro-oceans-and-atmosphere-information-and-data-centre%20OR%20organization%3Acsiro-marine-and-amospheric-research%20OR%20organization%3Astate-of-the-environment%20OR%20organization%3Anew-south-wales-datasets%20OR%20organization%3Asouth-australian-governments%20OR%20organization%3Amurray-darlingbasinauthority)&start=0

[mwu2018]

Upload the following json to v7 https://nationalmap.gov.au and see the harvested ckan catalog without organisation filetering:

  {
    "homeCamera": {
      "north": -8,
      "east": 158,
      "south": -45,
      "west": 109
    },
    "corsDomains": [
      "corsproxy.com",
      "programs.communications.gov.au",
      "mapsengine.google.com",
      "s3-ap-southeast-2.amazonaws.com",
      "data.melbourne.vic.gov.au",
      "data.act.gov.au",
      "dea.ga.gov.au",
      "gsky.nci.org.au",
      "discover.data.vic.gov.au"
    ],
    "catalog": [
      {
        "blacklist": {
          "Rainforests (EVC_2005)": true
        },
        "groupBy": "organization",
        "minimumMaxScaleDenominator": 10000000000,
        "filterByWmsGetCapabilities": false,
        "filterQuery": [
          {
            "fq": "+(res_format:(geojson OR GeoJSON OR kml OR KML OR kmz OR KMZ OR wms OR WMS OR CSV-GEO-AU OR csv-geo-au OR aus-geo-csv OR \"Esri REST\"))"
          }
        ],
        "url": "http://www.data.gov.au",
        "type": "ckan",
        "name": "Data.gov.au (No org filtering)",
        "includeCsv": true,
        "includeGeoJson": true,
        "includeKml": true,
        "includeWms": true,
        "includeWfs": true,
        "includeEsriMapServer": true,
        "includeEsriFeatureServer": true
      }
    ]
  }

[mwu2018]

Without organisation filtering, the number of harvested ckan data items is substantially larger than with filtering:

[AnaBelgun]

@mwu2018 thank you; that looks great. Is that a way to check what's under each group? can I test it somewhere? maybe the data from councils coming through is not usable.

[AnaBelgun]

@mwu2018 to continue investigation with:

• [ ] the issue of inconsistent search results based on quality ratings
• [ ] the need for a UI to accommodate magda backed search of dga datasets harvested