kafka-elasticsearch-injector
kafka-elasticsearch-injector copied to clipboard
inloco
Golang app to read records from a set of kafka topics and write them to an elasticsearch cluster
Bumps [github.com/olivere/elastic/v7](https://github.com/olivere/elastic) from 7.0.25 to 7.0.32. Release notes Sourced from github.com/olivere/elastic/v7's releases. Release 7.0.32 Update dependencies Update CI to Go 1.17 and Go 1.18 Add tracer for OpenTelemetry Add max_analyzed_offset...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Bumps [github.com/go-kit/kit](https://github.com/go-kit/kit) from 0.10.0 to 0.12.0. Release notes Sourced from github.com/go-kit/kit's releases. v0.12.0 A mostly procedural update to Go kit, including support for the latest version of Go, and updates...
Bumps [github.com/Shopify/sarama](https://github.com/Shopify/sarama) from 1.24.1 to 1.27.1. Release notes Sourced from github.com/Shopify/sarama's releases. Version 1.27.1 (2020-10-07) Improvements #1775 - @d1egoaz - Adds a Producer Interceptor example #1781 - @justin-chen - Refresh...
### Description I have added a deadline for the consumer to flush non-empty batches to the storage service, which defaults to one minute without receiving new records. This deadline is...
Bumps [github.com/olivere/elastic/v7](https://github.com/olivere/elastic) from 7.0.23 to 7.0.24. Commits 4cc7d49 Release 7.0.24 f3161e2 Update list of contributors and fix file name 685abfd Add Feature cat fielddata API (#1476) 2365c21 Add Point In...
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot] avatar"){kind=avatar}
Bumps [github.com/Shopify/sarama](https://github.com/Shopify/sarama) from 1.24.1 to 1.28.0. Release notes Sourced from github.com/Shopify/sarama's releases. Version 1.28.0 (2021-02-15) Note that with this release we change RoundRobinBalancer strategy to match Java client behavior. See...
I this following code may not work in `concurrent miss` scenario https://github.com/inloco/kafka-elasticsearch-injector/blob/master/src/kafka/encode_decode.go#L115-L142 the solution described here with _ChanLocker_ pattern might fix it. https://lakefs.io/2020/09/23/in-process-caching-in-go-scaling-lakefs-to-100k-requests-second/
https://github.com/bsm/sarama-cluster is now deprecated. We should replace it with https://github.com/Shopify/sarama
KAFKA_CONSUMER_BATCH_SIZE controls when the in-memory events are flushed to ES. However some topics may have small throughput, not having their events flushed at all. It would be nice to have...
