paimon
paimon copied to clipboard
apache
[Bug] parquet java version has a logging bug when debug logging is turned on
Search before asking
- [x] I searched in the issues and found nothing similar.
Paimon version
10.0
Compute Engine
JavaAPI
Minimal reproduce step
I'm just writing out data into s3 using a PostgreSQL catalog, and suddenly things start failing with a long stack trace which crashes out the thread:
2025-03-08T00:51:58.899543736Z ERROR Unable to format msg: 0, VisitedIndex{visitedIndexes={}}: [] r:0 java.lang.IllegalArgumentException: found 1 argument placeholders, but provided 0 for pattern `0, VisitedIndex{visitedIndexes={}}: [] r:0`
at org.apache.logging.log4j.message.ParameterFormatter.formatMessage(ParameterFormatter.java:248)
at org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:282)
at org.apache.logging.log4j.core.pattern.MessagePatternConverter$SimpleMessagePatternConverter.format(MessagePatternConverter.java:120)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44)
at org.apache.logging.log4j.core.pattern.StyleConverter.format(StyleConverter.java:117)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:44)
at org.apache.logging.log4j.core.pattern.HighlightConverter.format(HighlightConverter.java:248)
at org.apache.logging.log4j.core.layout.PatternLayout$NoFormatPatternSerializer.toSerializable(PatternLayout.java:355)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:252)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:238)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:58)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:227)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:220)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:211)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:160)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:133)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:124)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:88)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:705)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:663)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:639)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:575)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:92)
at org.apache.logging.log4j.core.Logger.log(Logger.java:169)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2906)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2859)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2841)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2631)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:2394)
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:128)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.log(MessageColumnIO.java:283)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.printState(MessageColumnIO.java:269)
at org.apache.paimon.shade.org.apache.parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.startMessage(MessageColumnIO.java:294)
at org.apache.paimon.shade.org.apache.parquet.io.RecordConsumerLoggingWrapper.startMessage(RecordConsumerLoggingWrapper.java:175)
at org.apache.paimon.format.parquet.writer.ParquetRowDataWriter.write(ParquetRowDataWriter.java:71)
at org.apache.paimon.format.parquet.writer.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:75)
at org.apache.paimon.format.parquet.writer.ParquetRowDataBuilder$ParquetWriteSupport.write(ParquetRowDataBuilder.java:56)
at org.apache.paimon.shade.org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:138)
at org.apache.paimon.shade.org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:103)
at org.apache.paimon.format.parquet.writer.ParquetBulkWriter.addElement(ParquetBulkWriter.java:47)
at org.apache.paimon.io.SingleFileWriter.writeImpl(SingleFileWriter.java:126)
at org.apache.paimon.io.StatsCollectingSingleFileWriter.write(StatsCollectingSingleFileWriter.java:84)
at org.apache.paimon.io.KeyValueDataFileWriter.write(KeyValueDataFileWriter.java:123)
at org.apache.paimon.io.KeyValueDataFileWriter.write(KeyValueDataFileWriter.java:58)
at org.apache.paimon.io.RollingFileWriter.write(RollingFileWriter.java:80)
at org.apache.paimon.mergetree.SortBufferWriteBuffer.forEach(SortBufferWriteBuffer.java:162)
at org.apache.paimon.mergetree.MergeTreeWriter.flushWriteBuffer(MergeTreeWriter.java:225)
at org.apache.paimon.mergetree.MergeTreeWriter.prepareCommit(MergeTreeWriter.java:264)
at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit(AbstractFileStoreWrite.java:218)
at org.apache.paimon.operation.MemoryFileStoreWrite.prepareCommit(MemoryFileStoreWrite.java:155)
at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:253)
at com.toro.trc.services.confluence.utils.TAvroToPaimon.writeToTable(TAvroToPaimon.java:137)
at com.toro.trc.services.confluence.sinks.TPaimonTableSink.publish(TPaimonTableSink.java:70)
at com.toro.trc.services.confluence.sinks.TRobotPulsarPaimonSink.publish(TRobotPulsarPaimonSink.java:51)
at com.toro.trc.services.confluence.TConflux.writeToSink(TConflux.java:146)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
What doesn't meet your expectations?
The thread crashes because of a log4j issue inside of the parquet library. Obviously not ideal.
Anything else?
Took a little bit, but I found that this is a known issue with parquet-java that has since been fixed in this PR: https://github.com/apache/parquet-java/pull/1294
The first release to include this fix was 1.14.0. So I assume just upgrading to at least that version should address the issue. Latest version is 1.15.0. Would it be reasonable to just move to the latest version?
Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
