knime-geospatial-extension icon indicating copy to clipboard operation
knime-geospatial-extension copied to clipboard

Add Socrata Data Nodes

Open UrbanGISer opened this issue 2 years ago • 3 comments

Socrata Search: To search the database and get the data list Socrata Data Query: get the data based on domain and ID。

UrbanGISer avatar Nov 14 '23 22:11 UrbanGISer

Test workflows can be found here: https://hub.knime.com/center%20for%20geographic%20analysis%20at%20harvard%20university/spaces/Workflow%20Tests/WIP~yP2rfc1sFgXgNhQz/

koettert avatar Feb 11 '25 15:02 koettert

@UrbanGISer just played around with the nodes and have some suggestions

  • Would it make sense/could we move the metadata.domain next to the resource.id column to the beginning of the table since these two info are needed in the Socrata Data Query node
  • We should add some progress information to the Socrata Data node. I guess we do not know in advanced how many rows will be returned so we cannot update the percentage progress. However what we can do is we can update the progress message with each iteration of the while True loop and include the current offset so that the user sees that the node is working hard on downloading the data
  • We should mention the throttling in the node description and that it can be avoided by creating an application token that can be provided via a new advanced setting
  • Does the Socrata Search node only return public data sets? If not can we restrict the result to only show public data sets
  • I think it would be also great if we could add an advanced query field (simple text field) to the Socrata Query Node that allows the user to specify an optional simple filter or query which gets simply appended as query parameter to the URL

koettert avatar Mar 28 '25 16:03 koettert

  • e.id colu

Updated accordingly

  • [x] Would it make sense/could we move the metadata.domain next to the resource.id column to the beginning of the table since these two info are needed in the Socrata Data Query node
  • [x] We should add some progress information to the Socrata Data node. I guess we do not know in advanced how many rows will be returned so we cannot update the percentage progress. However what we can do is we can update the progress message with each iteration of the while True loop and include the current offset so that the user sees that the node is working hard on downloading the data
  • [x] We should mention the throttling in the node description and that it can be avoided by creating an application token that can be provided via a new advanced setting
  • [x] Does the Socrata Search node only return public data sets? If not can we restrict the result to only show public data sets
  • [x] I think it would be also great if we could add an advanced query field (simple text field) to the Socrata Query Node that allows the user to specify an optional simple filter or query which gets simply appended as query parameter to the URL

UrbanGISer avatar May 02 '25 16:05 UrbanGISer

Merged via https://github.com/spatial-data-lab/knime-geospatial-extension/pull/415

koettert avatar Jul 02 '25 17:07 koettert