hudi
hudi copied to clipboard
apache
Upserts, Deletes And Incremental Processing on Big Data.
As discussed in [pull/5627|https://github.com/apache/hudi/pull/5627#discussion_r913165917], we may need to standardize the compare field and rename `orderingVal` to a more appropriate meaning ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-4358 - Type: Sub-task -...
org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter#testDFSTwoFilesWriteWithRollover {code:java} org.mockito.exceptions.verification.TooManyActualInvocations: avroFileDeltaInputWriter.canWrite(); Wanted 2 times: -> at org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119) But was 3 times: -> at org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50) -> at org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50) -> at org.apache.hudi.integ.testsuite.writer.DFSDeltaWriterAdapter.write(DFSDeltaWriterAdapter.java:50) at org.apache.hudi.integ.testsuite.TestDFSHoodieTestSuiteWriterAdapter.testDFSTwoFilesWriteWithRollover(TestDFSHoodieTestSuiteWriterAdapter.java:119) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ...
[https://github.com/apache/spark/blob/v3.5.2/connector/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala] [https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala] hudi version has not been kept up to date with spark ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-9723 - Type: Task - Fix version(s): - 1.1.0
## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-8892 - Type: New Feature
[https://github.com/apache/hudi/pull/13615/#discussion_r2277484016] ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-9719 - Type: Improvement - Fix version(s): - 1.1.0
A few TODOs: [https://github.com/apache/hudi/pull/13615/#discussion_r2274253729] [https://github.com/apache/hudi/pull/13615/#discussion_r2274271097] [https://github.com/apache/hudi/pull/13615/#discussion_r2274346707] [https://github.com/apache/hudi/pull/13615/#discussion_r2277390249] [https://github.com/apache/hudi/pull/13615/#discussion_r2277504954] [https://github.com/apache/hudi/pull/13615/#discussion_r2277511364] [https://github.com/apache/hudi/pull/13615/#discussion_r2277516019] [https://github.com/apache/hudi/pull/13615/#discussion_r2277524874] [https://github.com/apache/hudi/pull/13615/#discussion_r2277532615] [https://github.com/apache/hudi/pull/13615/#discussion_r2277550470] [https://github.com/apache/hudi/pull/13615/#discussion_r2277556629] [https://github.com/apache/hudi/pull/13615/#discussion_r2277599931] https://github.com/apache/hudi/pull/13615/#discussion_r2277685486 ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-9718 - Type: Improvement - Fix version(s): - 1.1.0
https://github.com/apache/hudi/pull/13615/#discussion_r2277473407 ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-9716 - Type: Improvement - Fix version(s): - 1.1.0
Incremental timestamps should support taking in `yyyyMMddHHmmssSSS`, or ISO timestamps like `2025-01-02` or `2025-01-02T03:04:56.789`, etc, or epoch seconds or mills. Currently, random numbers like 42 can be passed to it...
Since Hudi 1.x does not allow upgrade from and downgrade to table version lower than SIX, there are tests that are obsolete or no longer needed. These tests should be...
[https://github.com/apache/hudi/pull/13563#discussion_r2217009545] There is a need to control whether to write bloom filter and the number of entries per bloom filter per MDT partition to differentiate among partitions (FILES, COLUMN_STATS, RECORD_INDEX,...