logging-flume icon indicating copy to clipboard operation
logging-flume copied to clipboard

add redis channel

Open JunLuo opened this issue 8 years ago • 9 comments

In some cases, we need events can not be lost. But we don't want to install zk & kafka. To make event deliver only once and to deploy many flume agents in different nodes but share only one channel, we developed redis channel. We already use the redis channel in out production environment, and we want to feed back to flume.

JunLuo avatar Sep 16 '17 13:09 JunLuo

Hi @JunLuo ,

Thanks for the pull request!

This looks like a very useful change to me. However, Travis CI has reported a compilation error:

[ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[3,22] error: package junit.framework does not exist [ERROR] /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[4,22] error: package junit.framework does not exist [ERROR] /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[5,22] error: package junit.framework does not exist [ERROR] /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[11,12] error: cannot find symbol [ERROR] class TestCase /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[26,18] error: cannot find symbol [ERROR] class AppTest /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[28,19] error: cannot find symbol [ERROR] class AppTest /home/travis/build/apache/flume/flume-ng-channels/flume-redis-channel/src/test/java/org/flume/redis/channel/AppTest.java:[36,8] error: cannot find symbol [INFO] 7 errors

Can you please fix it, so that we can proceed with the reviews?

Thank you,

Donat

bessbd avatar Sep 16 '17 15:09 bessbd

Hi @bessbd , I have fixed the errors about Junit and code style :)

JunLuo avatar Sep 16 '17 23:09 JunLuo

Hi @JunLuo ,

Thank you for the changes!

On a first pass, I've noticed that a new user guide entry is missing. For this change to get merged, some documentation would be necessary. For an example, I'd recommend Kafka Channel. Do you think you can make the necessary additions?

Thank you,

Donat

bessbd avatar Sep 25 '17 21:09 bessbd

Hi @bessbd , I've add some documentation about redis channel :)

JunLuo avatar Sep 26 '17 10:09 JunLuo

Hi @szaboferee , I've done these changes:

  1. Add cluster and sentinel mode
  2. Remove useless optional
  3. Add the documention about cluster and sentinel
  4. Move redis config to flume configuration
  5. Fix the error of singleton
  6. Add unit test

JunLuo avatar Sep 29 '17 14:09 JunLuo

Hi @szaboferee , I've fix all problems you mentions in commits, What should I do next?

JunLuo avatar Oct 28 '17 02:10 JunLuo

Overall, this change looks fine to me. After receiving feedback for the comments I have left, I'd like to request contributors and committers to do a review of this PR so that we can merge it.

bessbd avatar Oct 29 '17 12:10 bessbd

Hi @bessbd , I've fix all problems you mentions in commits. :)

JunLuo avatar Nov 11 '17 03:11 JunLuo

Can one of the admins verify this patch?

asfgit avatar Aug 17 '18 13:08 asfgit