presto icon indicating copy to clipboard operation
presto copied to clipboard

Minor bug when querying hudi MoR tables

Open pratyakshsharma opened this issue 3 years ago • 0 comments

When trying to query Hudi table stored on S3 path with the new presto-hudi connector, I came across this error -

Query - select * from hudi.default.hudi_mor_rt;

org.apache.hudi.exception.HoodieException: Exception when reading log file at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:352) at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:192) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:110) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:103) at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:324) at org.apache.hudi.hadoop.realtime.RealtimeCompactedRecordReader.getMergedLogRecordScanner(RealtimeCompactedRecordReader.java:94) at org.apache.hudi.hadoop.realtime.RealtimeCompactedRecordReader.<init>(RealtimeCompactedRecordReader.java:64) at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.constructRecordReader(HoodieRealtimeRecordReader.java:70) at org.apache.hudi.hadoop.realtime.HoodieRealtimeRecordReader.<init>(HoodieRealtimeRecordReader.java:47) at org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat.getRecordReader(HoodieParquetRealtimeInputFormat.java:74) at com.facebook.presto.hudi.HudiRecordCursors.createRecordReader(HudiRecordCursors.java:116) at com.facebook.presto.hudi.HudiRecordCursors.lambda$createRealtimeRecordCursor$0(HudiRecordCursors.java:83) at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23) at com.facebook.presto.hive.HdfsEnvironment.doAs(HdfsEnvironment.java:81) at com.facebook.presto.hudi.HudiRecordCursors.createRealtimeRecordCursor(HudiRecordCursors.java:82) at com.facebook.presto.hudi.HudiPageSourceProvider.createPageSource(HudiPageSourceProvider.java:127) at com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:63) at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:80) at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:261) at com.facebook.presto.operator.Driver.processInternal(Driver.java:426) at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:309) at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:730) at com.facebook.presto.operator.Driver.processFor(Driver.java:302) at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1079) at com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:166) at com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599) at com.facebook.presto.$gen.Presto_null__testversion____20220803_082738_1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.UnsupportedOperationException: Not implemented by the PrestoS3FileSystem FileSystem implementation at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:219) at org.apache.hadoop.fs.HadoopExtendedFileSystem.getScheme(HadoopExtendedFileSystem.java:71) at org.apache.hudi.common.fs.FSUtils.isGCSFileSystem(FSUtils.java:628) at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:476) at org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:114) at org.apache.hudi.common.table.log.HoodieLogFormatReader.<init>(HoodieLogFormatReader.java:70) at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:219) ... 29 more

This needs to be fixed.

pratyakshsharma avatar Aug 11 '22 08:08 pratyakshsharma