apache
[SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant:[20220727185451835__commit__COMPLETED]
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.
I use java client hudi-0.11.0 version， when data is large, hudi client produces the above error, could anyone help?
@jtchen-study can you share the Hudi configs and the setup you use for Java client?
@yihua yes , thanks for your reply
public static HoodieWriteConfig buildCfg(String tablePath, String tableSchema, String tableName) { return HoodieWriteConfig.newBuilder().withPath(tablePath) .withSchema(tableSchema) .withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder() .withStorageType(FileSystemViewStorageType.MEMORY) .build()) .withEmbeddedTimelineServerEnabled(false) .withAutoCommit(true) .withWriteConcurrencyMode(WriteConcurrencyMode.SINGLE_WRITER) .withMarkersType(MarkerType.DIRECT.toString()) .withEngineType(EngineType.JAVA) .forTable(tableName) .withIndexConfig(HoodieIndexConfig.newBuilder() .withIndexType(HoodieIndex.IndexType.BLOOM).build()) .withCompactionConfig(HoodieCompactionConfig.newBuilder() .withAutoClean(false) .withAutoArchive(false) .build()) .build(); }
@yihua i am trying to develop a hudi writer plugin into DATAX， but i found that java client have some performance issue, besides the above one issue, do you have any suggession about perfermance enhance? as i know that java client does not support multi thread
The validation is legitimate, i.e. for a completed instant the marker dir is removed, so rollback cannot be done based on markers.
Just to go past this issue, you can disable hoodie.rollback.using.markers (https://hudi.apache.org/docs/configurations#hoodierollbackusingmarkers). To fix this properly, I would suggest that Hudi either falls back to ListingBasedRollbackStrategy in such cases, or fails fast (at the time of scheduling rollback itself so that users can disable rollback using markers before rollback plan is written). @yihua wdyt?
@jtchen-study Ideally, rollback is triggered only for failed writes. As such fallback to listing-based rollback should be safe but we need to understand how rollback got triggered for completed instant. Can you describe the sequence of events that happened and also the setup. Steps to reproduce would be very helpful. Is this a single-writer or multi-writer scenario? I've create HUDI-4550 to track the investigation.
We have put in a fix for this https://github.com/apache/hudi/pull/6313 Can you give that a try and let us know if it resolves the issue.
closing the issue as we have the fix with 0.12. thanks for reaching out. appreciate you keeping the community buzzing.
The validation is legitimate, i.e. for a completed instant the marker dir is removed, so rollback cannot be done based on markers. Just to go past this issue, you can disable
hoodie.rollback.using.markers(https://hudi.apache.org/docs/configurations#hoodierollbackusingmarkers). To fix this properly, I would suggest that Hudi either falls back toListingBasedRollbackStrategyin such cases, or fails fast (at the time of scheduling rollback itself so that users can disable rollback using markers before rollback plan is written). @yihua wdyt?
Does disabling 'hoodie.rollback.using.markers' make rollback inefficient?
