Build/String Query based on Record Count

[JJediny]

The ESRI JSON to GeoJSON hack used in concatenating the URL to MapServer/0/ + query?where=objectid+%3D+objectid&outfields=*&f=json is limited by the ArcGIS Server's MaxRecordCount (default=1000). As a result the response will only return the first { MaxRecordCount# } features; one way to find out if this is an issue is to add query?returnCountOnly=true&where=objectid+%3D+objectid&outfields=*&f=json which will provide a total count of all features for that layer.

Feature Consideration:

[ ] - Add test for response; so that {count} - {features returned} = 0 --> NO --> issue another sets of queries objectIds= 1001, 1002, 1003 (...) [ ] - Or tie the query to be issued from the BBox of the mapview; not async but with user triggering a resync/load.

Just some ideas after testing the plugin; great app though has made scrapping arc servers alot easier. Thanks

Alternatively you should be able to string together a query to request specific objectIds;

• Syntax: objectid=<objectId1>, <objectId2>
• query?where=objectid=501, 502&outSR=4326&outfields=*&f=json

Reference: http://jasonsanford.github.io/leaflet-vector-layers/demos/arcgis-server/ http://resources.arcgis.com/en/help/rest/apiref/query.html

[MatzFan]

I have just written a simple command line utility in Ruby to cycle through all records for a given layer and build a single JSON file - which can then be imported into QGIS via the 'Add Vector Layer' by protocol (JSON) menu. So far tested up to ~134k record layer. I may also extend it to build files for all sub-layers within a given layer, for bulk import.

You are welcome to adapt/use this if it is useful, or I could look to port it to Python & do a PR to your repo if you are interested?

[ajturner]

There are a lot of idiosyncrasies of ArcGIS server pagination depending on version, database used, primary key, deleted features, etc.

Here is a general Python library we (Esri) maintain https://github.com/esri/ArcREST

And a stand alone Node JS library https://github.com/koopjs/featureservice

[MatzFan]

Great, thanks

[sfkeller]

We've resolved that AFAIK including parallel workers to fetch the GeoJSON. See this Beta release: https://github.com/geometalab/ArcGISConnector-QGIS-Plugin/tree/beta Your're welcome to test it using manual installation of the plugin:

git clone --branch=beta https://github.com/geometalab/ArcGISConnector-QGIS-Plugin.git

Pls. be aware that this plugin only reads Feature Layers (from Mapserver and Feature Service). For reading Image Layers and other tile map services use e.g. QuickMapServices plugin: https://github.com/nextgis/quickmapservices

[MatzFan]

I may well do that, thanks Stefan - and for the pointer to the other plugin

[MatzFan]

Will try and test your Beta over the weekend Stefan. Just one question/feature suggestion: Can it dynamically generate sub-Groups in QGIS by any chance? - i.e. for Group Layers containing Feature Layers - that would be awesome. My command line tool can now traverse a Group Layer structure recursively & write the Feature Layer GeoJSON files to sub-directories mirroring that structure. No parallelism yet though :smiley:

[sfkeller]

I'm not sure if i understand you correctly but currently our plugin accepts only valid weblinks. It does not parse any sub-directories. We know that this would be convenient but it's nice to have for now.

[MatzFan]

Should have given an example: This URL is "Group Layer" type with 3 sub Group Layers, each of which has 2 Feature Layers. In such cases it would be great to be able to import the whole structure in one go, by selecting a single link. This would involve dynamically creating QGIS Groups and possibly Groups within Groups, to mirror the structure on the MapServer.

e.g:

Main Group (created in QGIS, selected by user before plugin URL selection)

• Sub Group 1 - dynamically created QGIS group, named as per JSON Group Layer name
  • Feature Layer 1
  • Feature Layer 2
• Sub Group 2 - dynamically created QGIS group, named as per JSON Group Layer name
  • Feature Layer 3
  • Features Layer 4
• Sub Group 3

etc...

Appreciate this is more involved, so please consider it a FR if functionality doesn't already exist.