Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connectivity issues in hbase-dev 1.3 #43

Open
live-wire opened this issue Mar 25, 2020 · 3 comments · May be fixed by #46
Open

Connectivity issues in hbase-dev 1.3 #43

live-wire opened this issue Mar 25, 2020 · 3 comments · May be fixed by #46

Comments

@live-wire
Copy link

live-wire commented Mar 25, 2020 •
edited
Loading

We use your hbase-dev image in our integration tests and are getting connectivity issues like Caused by: java.net.ConnectException: Connection refused.
The point of using a versioned image instead of latest was that it would not be updated :)

Could you please revert it back to the image that it was before your update? Or point me to a git commit hash that I can build from myself?

@HariSekhon
@live-wire
Copy link
Author

Hi @HariSekhon,

Could you please show some love 😢
Here is a detailed error log if that helps ( + link to the failing travis build).

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Wed Apr 08 03:21:06 CEST 2020, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68194: row 'sanity_check,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hbase_host,16020,1586308795817, seqNum=0

	at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:207)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
	at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
	at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
	at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
	at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:802)
	at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
	at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
	at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
	at org.apache.hadoop.hbase.client.Admin$tableExists$2.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.hbaseSanityCheck(ReplicatorHBasePipelineIntegrationTestRunner.groovy:379)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner$hbaseSanityCheck$0.callCurrent(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:161)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.verifyThatEnvIsReady(ReplicatorHBasePipelineIntegrationTestRunner.groovy:269)
	at com.booking.replication.it.hbase.ReplicatorHBasePipelineIntegrationTestRunner.setupSpec(ReplicatorHBasePipelineIntegrationTestRunner.groovy:195)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:153)
	at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:84)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:180)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:164)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:157)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:152)
	at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:97)
	at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:84)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:480)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:463)
	at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:76)
	at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:67)
	at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68194: row 'sanity_check,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hbase_host,16020,1586308795817, seqNum=0
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1242)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:226)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:331)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:394)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:203)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	... 4 more

@live-wire live-wire mentioned this issue Apr 9, 2020
Merged
@yuvi17
Copy link

yuvi17 commented Jun 16, 2020

@HariSekhon @live-wire +1
getting a similar issue while connecting to hbase via shell.
the host_name (hbase_host,16020,1586308795817 in this case and similar in mine) is not getting resolved and thus causing the timeout.

Can you please look into this?

Attached log when I try via hbase shell

2020-06-17 00:31:25,185 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.2.4, r67779d1a325a4f78a468af3339e73bf075888bac, 2020年 03月 11日 星期三 12:57:39 CST
Took 0.0028 seconds
hbase(main):001:0> create 'test', 'cf'

ERROR: can not resolve c6daf6046bd6,16000,1592333486399

For usage try 'help "create"'

bdevetak pushed a commit to bdevetak/Dockerfiles that referenced this issue Jan 28, 2021
expose hbase region server port 16020 to fix /HariSekhon/issues/43/
3353450
@bdevetak bdevetak linked a pull request Jan 28, 2021 that will close this issue
Open
@bdevetak
Copy link

@HariSekhon : Problem is that HBase RS port is not exposed. Fixed on a fork, created pull request: #46

If you can merge to older branches (1.3 and 1.4) and rebuild docker hub images that would be super helpful.

Thanks!

@bdevetak bdevetak mentioned this issue Jan 28, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

expose hbase region server port 16020 to fix /HariSekhon/Dockerfiles/… bdevetak/Dockerfiles
3 participants
@bdevetak @live-wire @yuvi17