linkedin
Flume and Avro use netty framework
I wan't to use Avro lib to push on Avro-flume log centraliser but i have an error on flume.
Flume can't decode avro message because it-s not an netty call.
ressources: netty.io
@sebglon,
I understand and appreciate your frustration.
From the Flume user guide (https://flume.apache.org/FlumeUserGuide.html):
For example, an Avro Flume source can be used to receive Avro events from Avro clients or other Flume agents in the flow that send events from an Avro sink.
So it would completely make sense for goavro to also implement the Avro messaging protocols. However, today it only encodes and decodes binary data, with Avro OCF file format support. However, I don't have the ability to work on that now. I'd be happy to discuss and perhaps collaborate on this issue with another developer who has time to spend working on this feature enhancement.
I have developed this feature for integration with docker logging driver. Can you help me to validate Pull request? For info: docker docker/docker#26042
Thanks for the PR! I'm traveling with family on vacation and will look at this when I get back to my computer.
Thanks for the PR! I'm traveling with family on vacation and will look at this when I get back to my computer.
Now there is a separate project that implements Avro RPC protocol to communicate with a Flume server: https://github.com/myzhan/avroipc
This project implements both protocol specifications:
- https://avro.apache.org/docs/current/spec.html#Protocol+Declaration
- https://flume.apache.org/releases/content/1.9.0/FlumeDeveloperGuide.html#developing-custom-components
I've used one of the past version of the project in prod for a long time and didn't have any problems with it. It was refactored for better support of specifications recently and work is still in the process but may be used anyway.
