hudi
hudi copied to clipboard
apache
[SUPPORT] Can't query Redshift rows even after downgrade from 0.10
Describe the problem you faced
After the upgrade to Hudi 0.10, I faced the https://github.com/apache/hudi/issues/4283 issue in my environment, so my AWS Glue tables were working fine on AWS Athena, but were returning 0 rows with Redshift Spectrum. After that, I downgraded the tables to version 2 using Hudi-CLI, and returned to Hudi 0.9. I immediately tried to query the tables again after the downgrade, but i kept the same behavior, no errors and no rows returned. But, after doing a new load on this table, some rows appeared on the Redshift Spectrum query, not all of them, basically, i'm seeing a different number of rows on Redshift Spectrum (way less rows) in comparison with Athena (the complete set of rows).
The strange thing is: The table i'm facing this error have lots of upserts on lots of partitions (> than 1000 partitions) per batch, usually the updated partitions are days on the last 2 months, but there is some days on other dates aswell. The most recent partitions, are with the number of records ok, and big difference occurs on the old ones.
Does Hudi only "fix" the rows of the Redshift Spectrum incompatibility when there is a new load on a specific partition? There is any quick workaround to fix it?
To Reproduce
Steps to reproduce the behavior:
- Upgrade AWS Glue table to Hudi 0.10
- Load the table using Hudi 0.10
- Downgrade AWS Glue table to Hudi 0.9 with Hudi-CLI
- Load the table using Hudi 0.9
- The row count from Athena to Redshift Spectrum does not match anymore.
Expected behavior
All the rows should be visible on redshift spectrum after the downgrade.
Environment Description
- Hudi version : 0.10/0.9
- Spark version : 3.1.1
- Hive version : 2.3.7-amzn-4
- Hadoop version : 3.2.1
- Storage (HDFS/S3/GCS..) : S3
- Running on Docker? (yes/no) : no
@nochimow this is likely caused by the same issue in https://issues.apache.org/jira/browse/HUDI-3056 where the timeline's time precision is not handled properly. But when you write data with 0.9.0, the old time precision was used then partial data is shown in redshift. While we can't troubleshoot any aws services as we're open-source, I'm adding this issue to the ticket to push up the priority.
@xushiyan I opened a ticket to AWS support about this too to try to get more priority too.
Can you confirm what is the Hudi behaviour in this case? If i write new data into a partition, all rows in this partition will be visible to Redshift again or just the updated/inserted rows?
@nochimow I can confirm due to time precision change in 0.10.0 there needs to be some fix for redshift. I can't confirm the exact behavior, since the behavior is under the influence of the bug reported in that JIRA, also this is in redshift which i can't verify with. Hope you understand. We just need to prioritize the fix and follow up aws support. Please close this if you don't have further questions.
Probably you might have to do a restore to a older commit. Can you give it a try. add a savepoint to a commit which was created w/ 0.9.0. and then trigger a restore to that savepointed commit. it should restore your entire table to a older snapshot. (its destructive operation though. something to keep in mind) essentially restore will delete all data files and timeline files from now until the savepoint to which you are looking to restore.
@nochimow : let us know how did it go. or if you need hudi-cli commands to do restore, let us know.
Hello, I did a workaround to fix this issue that i didnt need to restore my table. Apparently, after the hudi table downgrade to 0.9 when I do a Hudi upsert operation into a partition, the rows of the partition become visible for the Redshift Spectrum again, so i forced kind of a fake update for all the partitions in my table. I'm still trying to correct all the rows all my partitions. But 95% of my rows are already normal again, maybe the remaining 5% is something i'm still missing on my side that i'm currently checking.
So for now the status is that using 0.10.x of Hudi is incompatible with redshift spectrum? We are exploring using Hudi for our data lake and came across this very same problem which confused me a lot. I'll do the downgrade now.
@nsivabalan Considering that we seem to be at AWS mercy here, could there be an option to have some config flag which uses the old precision? Or is there some new features relying on this change to be present?
We have this chicken and egg problem with adopting dbt now. We want to use redshift spectrum for queries and dbt spark for transforms. (Which was added in 0.10) But that renders spectrum unusable.
Best regards, Jørgen
@nsivabalan Would not be useful to list kind of "Know Issues and Limitations" on each release? I think that it's not clear for the users that there is this kind of incompatibility with Redshift Spectrum on 0,10 onwards. For many users the Redshift integration is a core feature.
@xushiyan Is it possible to try to escalate this issue with AWS again? There is still no return from their side?
yes, makes sense. I will see where we can document this.
Can we add it here https://hudi.apache.org/releases/release-0.10.0#writer-side-improvements ?
@umehrot2 : hey, is there any progress on this end. Can we bump this again internally if possible.
Hey guys, is it possible to have a config to keep the compatibility? For example using the old time precision
@codope @nsivabalan do you have updates on this subject? We stuck on Hudi 0.9 because of this :(
@nochimow Thought you had opened an AWS Support ticket. If yes, any update from AWS Support? Curious to know if Amazon Redshift Spectrum team is planning to do something on this issue.
@pomaster Yes i did, Their reply was that the product team is aware of this issue, but there is no ETA to fix this yet.
@nochimow Thanks for the info! When was the last update from AWS Support on your ticket? We are planning to reach out to AWS Redshift engineering team for help. Don't want to waste their time if there has already been a recent status update (e.g. < 1 month). Can you share your AWS Support ticket #? You can e-mail me at: [email protected].
@nochimow @rubenssoto and others: Looks like hudi 0.10.0 is supported from the docs https://docs.aws.amazon.com/redshift/latest/dg/c-spectrum-external-tables.html#c-spectrum-column-mapping-hudi Can you folks give it a try and let us know if things are working well w/ hudi 0.10.0.
Even with AWS saying that only 0.10.0 is "supported", I did some compatibility tests with Hudi 0.10, 0.11 and 0.12. All versions worked fine, like it wasn't before. (Prior to that, any table with Hudi version > 0.9 was returning 0 rows on Redshift Spectrum) The only detail here is that the Redshift version must be with the patch >=169. (Got this requirement from the AWS support)