Manu

Sorry for my wrong understanding of `SubTasks`. Hudi splits the input data by partition+fileGroup and then writes these partitioned data with parallelism of `write.tasks`. The job write 2000+ files in...

Can you redesign the partitions? There are only 1G - 2G of data, but there are so many partitions.

https://hudi.apache.org/docs/troubleshooting#how-can-i-resolve-the-nosuchmethoderror-from-hbase-when-using-hudi-with-metadata-table-on-hdfs

> @xicm Are you using multi writer setup? Single writer

   The instant in the screenshot comes from another job, its name is not the same as the one in the stack trace.

The root cause is the deltacommit in MDT rollbacks the compaction instant(compaction in MDT is a deltacommit) in MDT. When a compaction commits, it will create a **inflight DeltaCommit** in...

https://hudi.apache.org/docs/metadata#deployment-model-b-single-writer-with-async-table-services If we enable async table service with MDT we should config a lock.

Maybe we should set default value of hoodie.datasource.compaction.async.enable to false or make the metadata table non-blocking . It's confusing to user that single writer needs a lock by default. @danny0405

This seems a bug of parquet-avro.

https://hudi.apache.org/docs/metadata#deployment-model-b-single-writer-with-async-table-services Lock is needed for asycn table service with MDT.