open-semantic-etl icon indicating copy to clipboard operation
open-semantic-etl copied to clipboard

Published 20 hours ago • verified publisher icon opensemanticsearch

Enhanced error handling for plugins

Open opensemanticsearch opened this issue 8 years ago • 5 comments

Implement enhanced error handling (fallback plugins and retry) for data enrichment or data analysis plugins:

There should be parameters for each extraction & analysis plugin in the process chain for retry and fallback to alternate plugins using alternate tools or methods.

F.e. despite Apache Tika can not parse a file, the Linux command "file" can find out the content type.

opensemanticsearch avatar Jun 24 '16 10:06 opensemanticsearch

Part done: The ETL tools will print not only HTTP error code but the full error message from Solr if something went wrong while posting data to Solr index for easier debugging of schema or errors.

opensemanticsearch avatar Jul 20 '16 17:07 opensemanticsearch

ETL plugins using microservices / REST-APIs will retry failed connections: https://github.com/opensemanticsearch/open-semantic-etl/issues/84

Mandalka avatar Apr 07 '19 18:04 Mandalka

Error status / message management in own function in etl.py.

Mandalka avatar Apr 08 '19 17:04 Mandalka

Entity extraction by Solr text tagger(s) now with separated error handling for each tagger using this new error_message function, so status & error messages are indexed.

Mandalka avatar Apr 08 '19 17:04 Mandalka

All ETL plugins, which use microservices / HTTP REST-APIs for analysis now waiting for services that are down/not loaded yet by enhanced HTTP exception handling, which additionally provides more detailed error messages.

Mandalka avatar Apr 08 '19 17:04 Mandalka