hudi
hudi copied to clipboard
apache
[SUPPORT] Archive operation always release lock on the timeline when try lock failed
Dear community,
I have discovered an issue when using Hudi.If multiple archive tasks run on COW table and set "hoodie.archive.automatic=false", it may cause data problems. If set hoodie.archive.automatic=true, then this issue will not occur.
And then I find that if the archive operation try lock failed, it will always release lock(if exist).
I suspect that this lock release operation may have affected other normal operations.
Perhaps the problem could be avoided by doing it this way?
Looking forward to your valuable suggestions.
Hudi version :0.14.0
archive itself holds an transaction lock, so we need to release it in any case, what is the wrong case you have ecountered?
@danny0405 Sorry, I don't remember the details of the problem (I will confirm with my colleagues and provide the results later), but based on the logic here, wouldn't it be better to attempt to release the lock only when acquiring it, otherwise is there a chance of mistakenly releasing the lock of other operations and causing problems?
otherwise is there a chance of mistakenly releasing the lock of other operations and causing problems?
That's reasonable, we should ensure the lock been acquired in the first place.
@danny0405 @Ytimetravel The behavior of method TransactionManager#endTransaction itself is correct, it would check whether the current lock is hold by itself before it unlock. However, there is a bug in HoodieTimelineArchiver because archiving itself is not a transaction and does not correspond to any instant in timeline. When an exception occurs, it might mistakenly deletes locks held by others.
@Ytimetravel Would you like to fix this issue?
@beyond1920 Yes, I am very willing to fix this issue.
This fix has already been tested and verified~
What is the general reason that the trasanction start of archival is failing?