PHOENIX-6491 Phoenix High Availability

[dbwong]

Phoenix High Availability https://issues.apache.org/jira/browse/PHOENIX-6491

[gjacoby126]

@dbwong - looks like we're getting a build failure from undeclared dependencies:

[INFO] --- maven-dependency-plugin:3.1.1:analyze-only (enforce-dependencies) @ phoenix-core --- [WARNING] Used undeclared dependencies found: [WARNING] org.apache.hadoop:hadoop-hdfs:jar:3.1.4:compile [WARNING] com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [WARNING] org.apache.curator:curator-client:jar:4.2.0:compile [WARNING] org.apache.curator:curator-framework:jar:4.2.0:compile [WARNING] org.apache.curator:curator-recipes:jar:4.2.0:compile [WARNING] org.apache.hadoop:hadoop-common:test-jar:tests:3.1.4:test [WARNING] org.apache.hadoop:hadoop-annotations:jar:3.1.4:compile [WARNING] org.apache.httpcomponents:httpcore:jar:4.4.4:compile

[gjacoby126]

@dbwong - Thanks, I think you've addressed all my previous comments. However, it looks like the recent pom changes to address the implicit dependencies caused a lot of build errors with references to HDFS. It appears to be references to the org.apache.hadoop.io and org.apache.hadoop.conf packages that are the problem.

[gjacoby126]

I think if the build failures are addressed and we get a clean test run (not counting the refCount flappers I'm addressing on another JIRA) this should be good to go, unless someone else has more comments.

[gjacoby126]

@dbwong - the last commits did fix the build errors, but the ParallelPhoenixUtilTest getAnyOfNonExceptionallyTimeoutTest is failing because it's not actually timing out. Since this is a unit test failure, the IT tests didn't get a chance to run so there may be other test failures to look out for.

[gjacoby126]

@dbwong - we got to ITs this time but the new HA tests had several timeouts (which caused big thread dumps)

[gjacoby126]

@dbwong @stoty - I tried twice to kick this PR's test run off again at https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/view/change-requests/job/PR-1430/ but both times it failed in about 38s with an error saying it couldn't apply the patch to master. GitHub doesn't say the patch needs to be rebased, however.

Still, @dbwong , it might be worth rebasing this PR and force pushing to your feature branch to see if that helps.

[stoty]

I've seen Yetus have trouble with multiple commits in a PR before. You may want to try to squash your changes. (Maybe in another PR, if you don't want to lose the history)

[gjacoby126]

[ERROR] org.apache.phoenix.jdbc.FailoverPhoenixConnectionIT.testConnectionWhenActiveZKRestarts Time elapsed: 80.644 s <<< ERROR! org.apache.phoenix.exception.PhoenixIOException: hconnection-0xe3608ca closed at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:140) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1917) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1868) at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:2131) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:656) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:544) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:540) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:535) at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:818) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:447) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:439) at org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:336) at org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:407) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:1007) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:993) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:538) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:520) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:519) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:507) at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:2150) at org.apache.phoenix.jdbc.HighAvailabilityTestingUtility.doTestBasicOperationsWithStatement(HighAvailabilityTestingUtility.java:553) at org.apache.phoenix.jdbc.HighAvailabilityTestingUtility.doTestBasicOperationsWithConnection(HighAvailabilityTestingUtility.java:538) at org.apache.phoenix.jdbc.FailoverPhoenixConnectionIT.testConnectionWhenActiveZKRestarts(FailoverPhoenixConnectionIT.java:327) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.ConnectionImplementation$LocalConnectionClosedException: hconnection-0xe3608ca closed at org.apache.hadoop.hbase.client.ConnectionImplementation.checkClosed(ConnectionImplementation.java:628) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:824) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:64) at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:70) at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:283) at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:255) at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1076) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1034) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1018) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1890) ... 34 more

[dbwong]

[ERROR] org.apache.phoenix.jdbc.FailoverPhoenixConnectionIT.testConnectionWhenActiveZKRestarts Time elapsed: 80.644 s <<< ERROR! org.apache.phoenix.exception.PhoenixIOException: hconnection-0xe3608ca closed at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:140) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1917) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1868) at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:2131) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:656) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:544) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:540) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:535) at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:818) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:447) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:439) at org.apache.phoenix.compile.FromCompiler.getResolverForMutation(FromCompiler.java:336) at org.apache.phoenix.compile.UpsertCompiler.compile(UpsertCompiler.java:407) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:1007) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableUpsertStatement.compilePlan(PhoenixStatement.java:993) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:538) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:520) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:519) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:507) at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:2150) at org.apache.phoenix.jdbc.HighAvailabilityTestingUtility.doTestBasicOperationsWithStatement(HighAvailabilityTestingUtility.java:553) at org.apache.phoenix.jdbc.HighAvailabilityTestingUtility.doTestBasicOperationsWithConnection(HighAvailabilityTestingUtility.java:538) at org.apache.phoenix.jdbc.FailoverPhoenixConnectionIT.testConnectionWhenActiveZKRestarts(FailoverPhoenixConnectionIT.java:327) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.ConnectionImplementation$LocalConnectionClosedException: hconnection-0xe3608ca closed at org.apache.hadoop.hbase.client.ConnectionImplementation.checkClosed(ConnectionImplementation.java:628) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:824) at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:64) at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:70) at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:283) at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:255) at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1076) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1034) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1018) at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1890) ... 34 more

Ya I've seen this issue and have recreated locally. Still uncertain what is going on spent some time on this. Dayjob has been busy with release issues. @gjacoby126