sydney horan

Results 6 comments of sydney horan

I added a workaround for this issue in my local fork of Hudi. Small tweaks to [HoodieAsyncService.java](https://github.com/sydneyhoran/hudi/blob/20f182d82e020ecd30fc1546ea0a4a6116276195/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncService.java#L128) and [HoodieMultiTableDeltaStreamer.java](https://github.com/sydneyhoran/hudi/blob/20f182d82e020ecd30fc1546ea0a4a6116276195/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java#L405) were required and now it's working as expected. The executor shutdown...

After more testing, I believe one more [code change](https://github.com/sydneyhoran/hudi/blob/bde3719226bade5bce204cdc0d16fb3874123e0d/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L744) is required to stop the async services in other threads when NoNewDataTerminationStrategy is reached. Otherwise the background threads remained alive and...

Just a quick heads up - I ran the latest master MultiTableDeltaStreamer without `hoodie.deltastreamer.transformer.class` config and I got a NullPointerException due to the .split() in line: ```java List transformerClassNameOverride =...

I am also looking forward to this PR being merged 😄

It seems to be related to splitting the string in [SparkKeyGenUtils.scala#L47](https://github.com/apache/hudi/blob/9288fdc456f9a4215d32908756a4ddaee18abfc4/hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/SparkKeyGenUtils.scala#L47) especially since it only happens with customkeygen and not timestamp keygen. And when DeltaSync.java calls getPartitionFields to write to...

@berniedurfee-renaissance this was what I changed to make it work in my fork https://github.com/sydneyhoran/hudi/commit/b1692c6ba3901d40b0523fe5226b5c5bff51ac7f, but I'm sure it's not the most ideal path forward :)