dig-etl-engine icon indicating copy to clipboard operation
dig-etl-engine copied to clipboard

Published 20 hours ago • verified publisher icon usc-isi-i2

Coordinate with NYU do handle multiple projects

Open saggu opened this issue 8 years ago • 4 comments

  • [ ] Dynamically assign a kafka topic as output for the NYU crawler
  • [ ] Have the NYU crawler output kafka topic configurable via a config file
  • [ ] When user requests NYU crawl, update the configuration and then start the crawler

saggu avatar Oct 18 '17 00:10 saggu

status

szeke avatar Nov 08 '17 07:11 szeke

ACHE's backend supports multiple projects. But it still needs to update its GUI.

GreatYYX avatar Dec 14 '17 03:12 GreatYYX

We can integrate it, then when they release a new version the GUI will also work properly

szeke avatar Dec 14 '17 06:12 szeke

Problems reported to Aecio, not a big change. Because myDIG doesn't communicate to ACHE backend directly, it only has a link to its GUI and a consumer which listens to a named Kafka topic. We've already had a basic version of integration, but it still needs changes and tests when new ACHE is ready. @szeke

GreatYYX avatar Dec 14 '17 06:12 GreatYYX