DataflowTemplates
DataflowTemplates copied to clipboard
GoogleCloudPlatform
[Pub/Sub to Elasticsearch] Fix error handling
This pull request contains many changes in custom ElasticsearchIO to be able to handle failed inserts. Now ElasticsearchIO.Write is not a terminal step but returns JSON with error message to be sent to error-output-topic.
Now failed inserts don't slow down the pipeline and the throughput remain stable.
It's a draft PR and additional refactoring can be required.
@Cherepushko can you rebase please? Also, can you add in the description and code comments we are removing invalid fields (with only dot as name) so Elasticsearch doesn't throw exception and slow the throughput considerably.
Also left a comment for adding unit test for the method that removes dots from k8s logs
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Thanks for contributing!
There have been code reviews here that were not addressed (and the same thing about the structure of the project changing) -- do we still want to work on this PR?
Hi team, I'm a Google Cloud TAM that has a customer that needs this issue to be addressed. They have 30+ projects that sends logs to ElasticSearch and this error constantly appears. They will need the Pub/Sub to Elasticsearch template to catch such errors and make necessary formatting changes so that all logs from Google Cloud are accepted on ElasticSearch side. Could you please reopen this issue?
Hi team, I'm a Google Cloud TAM that has a customer that needs this issue to be addressed. They have 30+ projects that sends logs to ElasticSearch and this error constantly appears. They will need the Pub/Sub to Elasticsearch template to catch such errors and make necessary formatting changes so that all logs from Google Cloud are accepted on ElasticSearch side. Could you please reopen this issue?
Hey @cuigle! It's been a while but could you let me know if your customer still needs this improvement? Our team at Elastic consider maintaining this template and we can pick up this issue. It would be helpful if you share more detailed information about the issues your customer 's been facing.
Thanks.
