hivetarget【hive streaming】 Opened a connection to metastore, current connections too much , which made pipeline failed

[dongbin86]

trafficstars

HIVE_05 - Hive Metastore error: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused) at org.apache.thrift.transport.TSocket.open(TSocket.java:187) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:414) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:234) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:179) at com.streamsets.pipeline.stage.destination.hive.HiveTarget$1.run(HiveTarget.java:251) at com.streamsets.pipeline.stage.destination.hive.HiveTarget$1.run(HiveTarget.java:245) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at com.streamsets.pipeline.stage.destination.hive.HiveTarget.initHiveMetaStoreClient(HiveTarget.java:244) at com.streamsets.pipeline.stage.destination.hive.HiveTarget.init(HiveTarget.java:195) at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:52) at com.streamsets.pipeline.configurablestage.DStage.init(DStage.java:40) at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:136) at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:104) at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:53) at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:158) at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:97) at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:72) at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:761) at com.streamsets.datacollector.execution.runner.common.AsyncRunner$4.call(AsyncRunner.java:147) at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:233) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.thrift.transport.TSocket.open(TSocket.java:182) ... 27 more

[dongbin86]

[metadaddy]

Hi @sumpan - I'm not sure this is a bug in the tutorials. Could you post the problem to the sdc-user Google Group or our community Slack channel? See https://streamsets.com/community/ for details.

[metadaddy]

I just remembered - you can't get to either of those. I'll have someone take a look here.

[kunickiaj]

Hive Streaming is not a supported (by Cloudera) feature on CDH. StreamSets requires CDH 5.4 or newer to connect to Hive Streaming.

What version of Hive/CDH are you running?

[dongbin86]

@kunickiaj yeah,I notice documents issue about "hive version should > 0.13 for supporting hive streaming",I select CDH 5.7.2 and 5.8.0 both ,but not work! I download the latest 2.4.0.0 release version ,nothing changed , my hive table with 64 buckets ,I find metastore connecting increase up to 120 and some more may be, during this time also some connection closed,but ,runing for while ,that exception mentioned above occur ，then this pipeline retry this procedure，and occur that exception again , original and processor before hivetarget can see record throughput stop at 1000 , and hivetarget nothing I can see

[dongbin86]

@metadaddy thx ,man

[kunickiaj]

What Hive version is running on your cluster? Keep in mind that using a newer client (SDC side) won't help if your hive installation in the cluster is not matching.

[dongbin86]

@kunickiaj hive-1.1.0+cdh5.8.2+660-1.cdh5.8.2.p0.5.src.rpm , and sdc side I use cdh5.8