presto-hbase-connector icon indicating copy to clipboard operation
presto-hbase-connector copied to clipboard

presto的hbase依赖目录只需要编译出来的那一个jar包么

Open scienceyang opened this issue 5 years ago • 10 comments

presto版本0.221,我将编译出来的jar包放入plugin/hbase目录下,配置好hbase相关信息后,运行show tables;是正常的,但是运行select的时候select name from demo;就报错 2019-06-28T17:59:05.865+0800 DEBUG query-execution-12 com.facebook.presto.execution.QueryStateMachine Query 20190628_095905_00002_x8g6u is FAILED 2019-06-28T17:59:05.865+0800 DEBUG Query-20190628_095905_00002_x8g6u-257 com.facebook.presto.execution.QueryStateMachine Query 20190628_095905_00002_x8g6u failed java.lang.AbstractMethodError at com.facebook.presto.split.SplitManager.getSplits(SplitManager.java:87) at com.facebook.presto.split.CloseableSplitSourceProvider.getSplits(CloseableSplitSourceProvider.java:51) at com.facebook.presto.sql.planner.SplitSourceFactory$Visitor.lambda$visitTableScan$0(SplitSourceFactory.java:126) at com.facebook.presto.sql.planner.SplitSourceFactory$Visitor.visitTableScan(SplitSourceFactory.java:131) at com.facebook.presto.sql.planner.SplitSourceFactory$Visitor.visitTableScan(SplitSourceFactory.java:102) at com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:185) at com.facebook.presto.sql.planner.SplitSourceFactory$Visitor.visitAggregation(SplitSourceFactory.java:222) at com.facebook.presto.sql.planner.SplitSourceFactory$Visitor.visitAggregation(SplitSourceFactory.java:102) at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:199) at com.facebook.presto.sql.planner.plan.InternalPlanNode.accept(InternalPlanNode.java:31) at com.facebook.presto.sql.planner.SplitSourceFactory.createSplitSources(SplitSourceFactory.java:84) at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createStreamingLinkedStages(SqlQueryScheduler.java:411) at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createStreamingLinkedStages(SqlQueryScheduler.java:506) at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createStages(SqlQueryScheduler.java:327) at com.facebook.presto.execution.scheduler.SqlQueryScheduler.(SqlQueryScheduler.java:207) at com.facebook.presto.execution.scheduler.SqlQueryScheduler.createSqlQueryScheduler(SqlQueryScheduler.java:154) at com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:490) at com.facebook.presto.execution.SqlQueryExecution.startExecution(SqlQueryExecution.java:359) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 除了编译出的那两包还需要其他的jar包么

scienceyang avatar Jun 28 '19 09:06 scienceyang

需要把connector依赖的jar包都放到plugin/hbase目录下 也可以合起来打成一个比较大的jar包放到这个目录下,合起来打包的命令是: mvn -U clean package

Crossoverrr avatar Jun 28 '19 10:06 Crossoverrr

然而并不知道还依赖哪些jar。。

scienceyang avatar Jun 28 '19 10:06 scienceyang

现在查询存在这种情况 image 一个select * from course就一直卡在这里,debug日志也没有报错

scienceyang avatar Jun 28 '19 10:06 scienceyang

看一下worker有异常日志吗?

Crossoverrr avatar Jun 28 '19 10:06 Crossoverrr

知道啥问题了,zookeeper地址错了。。。打扰了

scienceyang avatar Jun 28 '19 10:06 scienceyang

OK,那我先close了啊。再有问题再打开

Crossoverrr avatar Jun 28 '19 10:06 Crossoverrr

你好,我运行mvn -U clean package之后报同样的错误。 prestodb:0.228 connector:0.221

0ZhangJc0 avatar Aug 05 '20 07:08 0ZhangJc0

我觉得这个问题应该是presto版本的问题,我是基于prestosql-315版本进行研发的,而我看你的日志应该是prestodb的,两者是不兼容的。如果你要基于prestodb来使用,可以考虑尝试dev_prestodb-0.221_0.1.2分支的代码。

Crossoverrr avatar Aug 05 '20 07:08 Crossoverrr

我觉得这个问题应该是presto版本的问题,我是基于prestosql-315版本进行研发的,而我看你的日志应该是prestodb的,两者是不兼容的。如果你要基于prestodb来使用,可以考虑尝试dev_prestodb-0.221_0.1.2分支的代码。

嗯嗯。应该是版本不兼容的问题。我是使用dev_prestodb-0.221_0.1.2分支的代码生成的connector,但是prestodb-0.228比prestodb-0.221的ConnectorPageSource接口多了getCompletedPositions()这个函数,所以才会发生上述问题

0ZhangJc0 avatar Aug 05 '20 08:08 0ZhangJc0

忘记贴报错信息了

2020-08-05T08:21:25.549Z ERROR remote-task-callback-28 com.facebook.presto.execution.StageExecutionStateMachine Stage execution 20200805_082125_00012_p5bbm.1.0 failed java.lang.AbstractMethodError: Method com/analysys/presto/connector/hbase/query/HBaseUpdatablePageSource.getCompletedPositions()J is abstract at com.analysys.presto.connector.hbase.query.HBaseUpdatablePageSource.getCompletedPositions(HBaseUpdatablePageSource.java) at com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:287) at com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:236) at com.facebook.presto.operator.Driver.processInternal(Driver.java:379) at com.facebook.presto.operator.Driver.lambda$processFor$8(Driver.java:283) at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:675) at com.facebook.presto.operator.Driver.processFor(Driver.java:276) at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077) at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162) at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:483) at com.facebook.presto.$gen.Presto_0_228_SNAPSHOT_6b48d04____20200805_074502_1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

0ZhangJc0 avatar Aug 05 '20 08:08 0ZhangJc0