publish to maven central repo && using elasticsearch-jdbc as a library

[abo]

Hi, I want call elasticsearch-jdbc in my application like this: new JDBCImporter().setSettings(settings).run();

then it will be more flexible, e.g. add a web ui

[ddbullfrog]

vote

[anandp504]

@jprante We need the jdbc import as part of the application code base instead of being a shell script. Is there a way we can use this as a dependency library and write java/scala code to perform the jdbc import?

[jprante]

Sure, here is a gradle example:

repositories {
    mavenLocal()
    mavenCentral()
    jcenter()
    maven {
        url "http://xbib.org/repository"
    }
}

apply plugin: 'maven'

dependencies {
   compile 'org.xbib.elasticsearch.importer:elasticsearch-jdbc:2.3.1.0'
}

[abo]

there are some static fields in JDBCImporter, StandardContext, StandardSource... so we can't start multi jdbcimporter instances in parallel

[jprante]

This will change with the next feature release.

[anandp504]

@jprante I get java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager exception when the library is used as part of the application. Do I have to add any other dependencies other than elasticsearch-jdbc?

override def indexCampaignData() = {
  val campaignsJdbcImporter = {
    val sql = s"""SELECT id, name, label from campaigns;"""
    val settings: Settings = Settings.settingsBuilder()
      .put("jdbc.url", Config.workflow_db_url)
      .put("jdbc.user", Config.workflow_db_username)
      .put("jdbc.password", Config.workflow_db_password)
      .put("jdbc.sql", sql)
      .put("jdbc.index", "campaigns")
      .put("jdbc.type", "campaigns")
      .put("jdbc.elasticsearch.host", "localhost")
      .put("jdbc.elasticsearch.port", 9300)
      .put("jdbc.elasticsearch.cluster", "visto-search")
      .build()
    new JDBCImporter().setSettings(settings)
  }
  campaignsJdbcImporter.run()
}

java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
    at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:498) ~[play_2.11-2.4.3.jar:2.4.3]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105) ~[play_2.11-2.4.3.jar:2.4.3]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:105) ~[play_2.11-2.4.3.jar:2.4.3]
    at play.utils.Threads$.withContextClassLoader(Threads.scala:21) ~[play_2.11-2.4.3.jar:2.4.3]
    at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:104) ~[play_2.11-2.4.3.jar:2.4.3]
Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
    at org.xbib.pipeline.AbstractPipeline.<clinit>(AbstractPipeline.java:19) ~[elasticsearch-jdbc-2.3.1.0.jar:na]
    at utils.ElasticSearchService.indexCampaignData(SearchService.scala:214) ~[classes/:na]
    at controllers.reports.SearchController$$anonfun$reindexCampaignsData$1.apply(SearchController.scala:139) ~[classes/:na]
    at controllers.reports.SearchController$$anonfun$reindexCampaignsData$1.apply(SearchController.scala:138) ~[classes/:na]
    at play.api.mvc.Action$.invokeBlock(Action.scala:533) ~[play_2.11-2.4.3.jar:2.4.3]
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.LogManager
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372) ~[na:1.8.0_20]
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.8.0_20]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_20]
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360) ~[na:1.8.0_20]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_20]

[eladtamary]

I had the same NoClassDefFoundError issue. Adding log4j-to-slf4j bridge fixed it.

<dependency>
    <groupId>org.xbib.elasticsearch.importer</groupId>
    <artifactId>elasticsearch-jdbc</artifactId>
    <version>2.3.1.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-to-slf4j</artifactId>
    <version>2.6.1</version>
</dependency>

[anandp504]

@eladtamary I still get the same error even after adding the dependency specified. Is there any other solution?

[erikaoctavia]

Is the maven dependency still available? Maven doesn't seem to recognize any of the versions.

<dependency> <groupId>org.xbib.elasticsearch.importer</groupId> <artifactId>elasticsearch-jdbc</artifactId> <version>2.3.4.1</version> </dependency>

[jprante]

JDBC importer is not ready to be used as a library, there are too many bugs. It's not on Maven central repository.