embulk-output-elasticsearch icon indicating copy to clipboard operation
embulk-output-elasticsearch copied to clipboard
verified publisher icon embulk

org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException

Open adrianovieira opened this issue 4 years ago • 3 comments

This error occurs after importing some data to elasticsearch

  • environment

    OS: Linux Fedora-33
elasticsearch: 7.11.2
ssl: yes; self-signed certificate
java: 
  openjdk version "1.8.0_265"
  OpenJDK Runtime Environment (Zulu 8.48.0.53-CA-linux64) (build 1.8.0_265-b11)
  OpenJDK 64-Bit Server VM (Zulu 8.48.0.53-CA-linux64) (build 25.265-b11, mixed mode)

  • Embulk and plugins

    Started Embulk v0.9.23
Loaded plugin embulk-input-postgresql (0.11.1)
Loaded plugin embulk-output-elasticsearch (0.4.7)
Loaded plugin embulk-filter-column (0.7.1)

  • logging: 

    021-03-17 16:08:24.241 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
021-03-17 16:08:37.086 -0300 [INFO] (0025:task-0000): > 12,85 seconds
021-03-17 16:08:37.088 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
021-03-17 16:08:37.089 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
021-03-17 16:08:37.090 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
021-03-17 16:08:37.093 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
021-03-17 16:08:37.096 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
021-03-17 16:08:37.098 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
021-03-17 16:08:37.191 -0300 [INFO] (0025:task-0000): > 0,09 seconds
021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
021-03-17 16:08:37.192 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
021-03-17 16:08:37.193 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
021-03-17 16:08:37.195 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
021-03-17 16:08:37.197 -0300 [INFO] (0025:task-0000): Fetched 8.000 rows.
021-03-17 16:08:37.199 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
021-03-17 16:08:37.255 -0300 [INFO] (0025:task-0000): > 0,06 seconds
021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 500 rows.
021-03-17 16:08:37.256 -0300 [INFO] (0025:task-0000): Fetched 1.000 rows.
021-03-17 16:08:37.257 -0300 [INFO] (0025:task-0000): Fetched 2.000 rows.
021-03-17 16:08:37.259 -0300 [INFO] (0025:task-0000): Fetched 4.000 rows.
021-03-17 16:08:37.260 -0300 [INFO] (0025:task-0000): SQL: FETCH FORWARD 10000 FROM cur
021-03-17 16:08:37.261 -0300 [INFO] (0025:task-0000): > 0,00 seconds
021-03-17 16:08:37.263 -0300 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
rg.embulk.exec.PartialExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
aused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:164)
at org.embulk.output.elasticsearch.ElasticsearchHttpClient.sendRequest(ElasticsearchHttpClient.java:322)
at org.embulk.output.elasticsearch.ElasticsearchHttpClient.push(ElasticsearchHttpClient.java:84)
at org.embulk.output.elasticsearch.ElasticsearchRecordBuffer.bufferRecord(ElasticsearchRecordBuffer.java:68)
at org.embulk.base.restclient.RestClientPageOutput.add(RestClientPageOutput.java:44)
at org.embulk.spi.PageBuilder.doFlush(PageBuilder.java:214)
at org.embulk.spi.PageBuilder.flush(PageBuilder.java:219)
at org.embulk.spi.PageBuilder.addRecord(PageBuilder.java:198)
at org.embulk.filter.column.ColumnFilterPlugin$1.add(ColumnFilterPlugin.java:237)
at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:351)
at org.embulk.exec.LocalExecutorPlugin$ScatterTransactionalPageOutput$OutputWorker.call(LocalExecutorPlugin.java:291)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
	... 16 more
Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
	at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
	at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
	at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
	... 14 more
Caused by: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
	at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
	at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	... 1 more
Caused by: java.io.IOException: Pipe quebrado
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
	... 7 more
Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
	... 16 more
Caused by: java.util.concurrent.ExecutionException: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
	at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
	at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
	at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
	at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
	... 14 more
Caused by: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:765)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)
	at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:364)
	at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:97)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	... 1 more
Caused by: java.io.IOException: Pipe quebrado
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170)
	... 7 more
aused by: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:228)
at org.embulk.util.retryhelper.jetty92.StringJetty92ResponseEntityReader.getResponse(StringJetty92ResponseEntityReader.java:31)
at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper$1.call(Jetty92RetryHelper.java:107)
at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
at org.embulk.util.retryhelper.jetty92.Jetty92RetryHelper.requestWithRetry(Jetty92RetryHelper.java:95)
... 14 more
aused by: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:277)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1309)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:182)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:129)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:89)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:122)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
... 1 more

rror: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.EOFException: HttpConnectionOverHTTP@78bbaf5(l:/10.0.1.54:47886 <-> r:elasticsearch-node01/10.0.0.17:9200,closed=false)[HttpChannelOverHTTP@5fee0e3c(exchange=HttpExchange@6a8f96a5 req=PENDING/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@37b18e38(req=HEADERS,snd=SENDING,failure=null)[HttpGenerator{s=COMPLETING}],recv=HttpReceiverOverHTTP@6d37c629(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of 0}]]]

adrianovieira avatar Mar 17 '21 19:03 adrianovieira

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

  • Did you import your certificate into the JVM environment? (i.e use keytool.)
  • Have you ever tried import without SSL I would like to know whether this issue relates to SSL or not.

Best regards.

hiroyuki-sato avatar May 12 '21 12:05 hiroyuki-sato

Hello, @adrianovieira

I'm not familiar with much Elasticsearch. Could you tell me the following?

  • Did you import your certificate into the JVM environment? (i.e use keytool.)

yes, I did it (if I hadn't done, the issue would be a certificate related).

  • Have you ever tried import without SSL

yes, on a "single-node" lab cluster! but it doesn't work for the production cluster which has more than one node and is over SSL.

I would like to know whether this issue relates to SSL or not.

Best regards.

adrianovieira avatar May 12 '21 13:05 adrianovieira

Hello, @adrianovieira Thank you for your reply.

  1. I'll try to create the same environment. I'll appreciate it if you send me the reproduce configuration of the elasticsearch.
  2. Multiple clusters without SSL worked fine. Correct?

hiroyuki-sato avatar May 13 '21 12:05 hiroyuki-sato