Elasticsearch5 sink

[geoffroyvergne]

Hi Flume team, I worked for my company on this patch to be able to use ElasticSearch 5 with flume. We heavy use Flume with Hadoop and ElasticSearch as well. We are testing this patch that seams to work properly I thought it could be helpfull for some of your users

Let me know if you have any questions

Geoffroy

[liorze]

Hi @geoffroyvergne, thank you for your contribution.

Can you please clean up your code a bit? i.e. minimize the changeset (remove comments and whitespace changes).

[zouyonghao]

I think here you should use

assertEquals("Coordinated Universal Time",
        factory.fastDateFormat.getTimeZone().getDisplayName(Locale.ENGLISH));

in the TestElasticSearchIndexRequestBuilderFactory.java

And additionly, I'm not sure whether you see my email that replied in the dev maillist.

[zouyonghao]

I put my patch for this on https://issues.apache.org/jira/browse/FLUME-3021

I fixed all unit tests and it should work. But need more tests. And it seems it need some configuration files like some in your pr, you can feel free to test this patch and add your code to this.

Thanks.

[geoffroyvergne]

Hi there, As you advised, I clean code (remove unusedd comments, and white spaces) And i fixed timezones assertions in the tests

BR

[liorze]

Thanks. Can you please also enable the checkstyle plugin (uncomment it in the pom file)?

We will also have to merge this one first https://reviews.apache.org/r/53056/, as elasticsearch 5.x requires Java 8.

[zouyonghao]

Hi, It seem all tests in TestElasticSearchSink could not be passed bacause the createNodes method do nothing. The solution is trying to mock the node. You can take a look at https://reviews.apache.org/r/57538/ I fixed all tests. Or maybe you can merge my commits at https://github.com/zouyonghao/flume the trunk branch. But I'm not sure whether to create a new pr or just use your pr, and whether you should post your patch at jira or just use github. I'm just a newbee to open source project.

Thanks.

[geoffroyvergne]

Hi, If you want to pass all tests, you need to have an ElasticSearch instance running in local. The standalone node doesn't exists anymore on the release 5.x There is a maven dependency test

org.elasticsearch.test framework 5.2.1

But I didn't implement it yet I'll try also to take a look at your patch

cheers

[zouyonghao]

But the unit tests are not integration tests, should be passed without any dependencies. And the embbed server has removed in es 5.x. So in my patch, I mock a node that can be used by the openLocalDiscoveryClient method which is defined in ElasticSearchTransportClient.

[bearrito]

How does this compare to https://issues.apache.org/jira/browse/FLUME-3021

[asfgit]

Can one of the admins verify this patch?

[rgoers]

Support for Elasticsearch has been removed from the main Flume build since it is so out of date and its new license is problematic. Please work against the flume-search repo.