hudi
hudi copied to clipboard
apache
Upserts, Deletes And Incremental Processing on Big Data.
Concerned about performance. How long should the following mocked-up sample take to write to s3? There are 1,369,765 records and 308 columns. It is taking ~10.5min running in docker container...
The current development route is that the next release will add some new functions and fix bugs in the old branch. However, the newly added functions will introduce new bugs....
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you...
``` 2022-07-18 16:49:53 org.apache.hudi.exception.HoodieIOException: Could not load filesystem view storage properties from hdfs://XXXXXX/user/tdw/warehouse/csig_billing_rt_ods.db/ods_dev_flow_t_operation_flow_ri/.hoodie/.aux/view_storage_conf.properties at org.apache.hudi.util.ViewStorageProperties.loadFromProperties(ViewStorageProperties.java:78) at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:213) at org.apache.hudi.util.StreamerUtil.getHoodieClientConfig(StreamerUtil.java:152) at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:376) at org.apache.hudi.util.StreamerUtil.createWriteClient(StreamerUtil.java:360) at org.apache.hudi.sink.compact.CompactFunction.open(CompactFunction.java:81) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) at...
```2022-07-19 05:44:23 org.apache.hudi.exception.HoodieIOException: Could not read commit details from hdfs://XXXXXX/.hoodie/20220719053423274.deltacommit at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.readDataFromPath(HoodieActiveTimeline.java:763) at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.getInstantDetails(HoodieActiveTimeline.java:264) at org.apache.hudi.common.table.timeline.HoodieDefaultTimeline.getInstantDetails(HoodieDefaultTimeline.java:372) at org.apache.hudi.hadoop.utils.HoodieInputFormatUtils.getCommitMetadata(HoodieInputFormatUtils.java:511) at org.apache.hudi.sink.partitioner.profile.WriteProfiles.getCommitMetadata(WriteProfiles.java:194) at org.apache.hudi.source.IncrementalInputSplits.lambda$inputSplits$71(IncrementalInputSplits.java:183) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)...
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you...
Hello guys. I am facing an issue on querying Data in Hudi version 0.10.1 using AWS glue. It works fine with 100 partitions in Dev but it got memory issues...
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you...
**Describe the problem you faced** We have some IoT data tables with a few thousands of partitions; typically `deviceId/year/month/day`. We do not sync to hive every commit, but at regular...
We upgraded ourselves from running our Hudi spark-submits from EMR 5.33 to EMR 6.5 that has Spark 3x and then started running into below errors with date and timestamp. Please...
