Connectivity issues in hbase-dev 1.3

[live-wire]

We use your hbase-dev image in our integration tests and are getting connectivity issues like Caused by: java.net.ConnectException: Connection refused. The point of using a versioned image instead of latest was that it would not be updated :)

Could you please revert it back to the image that it was before your update? Or point me to a git commit hash that I can build from myself?

@HariSekhon

[live-wire]

Hi @HariSekhon,

Could you please show some love 😢 Here is a detailed error log if that helps ( + link to the failing travis build).

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Wed Apr 08 03:21:06 CEST 2020, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68194: row 'sanity_check,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hbase_host,16020,1586308795817, seqNum=0

	at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
	at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
	at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
	at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
	at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)
	at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
	at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
	at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
	at org.apache.hadoop.hbase.client.Admin$tableExists$2.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.hbaseSanityCheck(ReplicatorHBasePipelineIntegrationTestRunner.groovy:379)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner$hbaseSanityCheck$0.callCurrent(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:161)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.verifyThatEnvIsReady(ReplicatorHBasePipelineIntegrationTestRunner.groovy:269)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.setupSpec(ReplicatorHBasePipelineIntegrationTestRunner.groovy:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:153)
	at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:84)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:180)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:164)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:157)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:152)
	at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:97)
	at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:84)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:76)
	at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:67)
	at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68194: row 'sanity_check,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hbase_host,16020,1586308795817, seqNum=0
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1242)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:394)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 4 more

[yuvi17]

@HariSekhon @live-wire +1 getting a similar issue while connecting to hbase via shell. the host_name (hbase_host,16020,1586308795817 in this case and similar in mine) is not getting resolved and thus causing the timeout.

Can you please look into this?

Attached log when I try via hbase shell

2020-06-17 00:31:25,185 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.2.4, r67779d1a325a4f78a468af3339e73bf075888bac, 2020年 03月 11日 星期三 12:57:39 CST
Took 0.0028 seconds
hbase(main):001:0> create 'test', 'cf'

ERROR: can not resolve c6daf6046bd6,16000,1592333486399

For usage try 'help "create"'

[bdevetak]

@HariSekhon : Problem is that HBase RS port is not exposed. Fixed on a fork, created pull request: https://github.com/HariSekhon/Dockerfiles/pull/46

If you can merge to older branches (1.3 and 1.4) and rebuild docker hub images that would be super helpful.

Thanks!