Epic 1: Data Harvesting & Integration

[anuveyatsu]

This epic encompasses the development of comprehensive data harvesting capabilities for PortalJS, enabling integration with multiple data catalog platforms.

Sub-Issues

This epic includes the following components:

• [x] #1432 Issue 1.1: CKAN Catalog Harvesting
• [x] #1427 Issue 1.2: Socrata Open Data Harvesting
• [x] #1428 Issue 1.3: OpenDataSoft (ODS) Harvesting
• [x] #1429 Issue 1.4: ArcGIS Hub/Portal Harvesting
• [x] #1430 Issue 1.5: Dataverse Repository Harvesting
• [ ] #1431 Issue 1.6: Custom Catalog Harvesting Framework
• [x] #1433 Issue 1.7: DKAN Harvesting
• [ ] https://github.com/datopian/portaljs/issues/1453 Issue 1.8: DKAN Harvesting

Goals

• Expand PortalJS data source coverage
• Provide standardized harvesting interfaces
• Enable automated data catalog synchronization
• Support multiple metadata schemas and formats
• Create extensible framework for custom integrations

Progress

2 of 7 issues completed (29%)

[demenech]

Hey @anuveyatsu

I would like to suggest having a DCAT harvester as well. We'd get great interoperability with a number of sources.

For example, if I wanted to harvest data from the Michigan Socrata portal, I could use the DCAT Harvester with https://data.michigan.gov/data.json instead of the Socrata Harvester.

[anuveyatsu]

@demenech yes, makes sense. Can you create a new issue for that and link from this epic?