Shawn Chang

Created this [gist](https://gist.github.com/CTTY/71c0f15384c390ef96b8382f9fa7b5ea) to track existing usages of FS based listing

Yes, this is related to the avro version change and this recent change of utilizing LTZ as well: https://github.com/apache/hudi/pull/6607/files Instead of reverting this [change](https://github.com/apache/hudi/pull/6628) directly, I think it makes more...

@mansipp is this similar to what we saw when reading Hudi tables via Trino?

I think we need to understand why Spark is trying to delete Hudi jars here, can you share the previous log so we can see what Spark is doing before...

This looks similar to this issue: https://github.com/apache/hudi/issues/7487 where user ran into S3 throttling issue due to too many S3 calls. Was wondering if you can check if there are too...

If you have to use Spark 3.5.0 then you can try cherry-picking this PR: #9717 Or you can downgrade Spark version used in your application to 3.3.0 to avoid conflicts....

I used to run into this issue and reverting this PR seems to work for me: #10254 Haven't got a chance to find the root cause though. You can try...

LGTM in general, could you fix the checkstyle issue as well? ``` src/main/java/org/apache/hudi/utilities/streamer/ErrorTableUtils.java:[38,8] (imports) UnusedImports: Unused import - java.lang.reflect.InvocationTargetException. ``` cc: @yihua

#12977 seems to be able to fix this issue, I have not tested it yet. I'll test it later

This issue exists on the released Hudi 1.0.2 as well and will be needed