[core] Support commit isolation level

[BsoBird]

Purpose

For the file system catalog, there are many situations where we need to support stricter commit policies.This is especially true for third-party databases that use paimon as their underlying storage.

We define the commit isolation level option to control the behavior.

Linked issue: none

1. Disable retry after failed commits in strict mode
2. In strict mode, reject and clean up potentially dirty commits.
3. Categorized possible exceptions and adjusted the exception handling logic.
4. Supports concurrency control.
5. In strict mode, do not use any hint file.

Tests

See FileStoreCommitTest.java.

API and Format

None

Documentation

Glossary:

• Dirty-commit: commitSnapshotId < CurrentLatestSnapshotId - snapshot.num-retained.max && commit-success=True && snapshotManager.exists(commitSnapshotId)=True && (commitSnapshotId not belong any tag/branch)

Flow chat (use strict mode):

    graph TD
START[startCommit] -->USE-LOCK{USE-LOCK?}
USE-LOCK -->|YES| COMMIT_WITH_LOCK{COMMIT_WITH_LOCK_SUCCESS}
USE-LOCK -->|NO| CHECK-BEFORE{CHECK-BEFORE-SUCCESS?}
CHECK-BEFORE -->|NO| COMMIT_FAILED
CHECK-BEFORE -->|YES| COMMIT{COMMIT_SUCCESS?}
COMMIT --> |commit state unknown| COMMIT_FAILED
COMMIT --> |not success| COMMIT_FAILED
COMMIT --> |YES| IS_DIRTY_COMMIT{IS_DIRTY_COMMIT?}
IS_DIRTY_COMMIT --> |YES| COMMIT_FAILED
IS_DIRTY_COMMIT --> |NO| COMMIT_SUCCESS
COMMIT_WITH_LOCK --> |failed-normal| RETRY
COMMIT_WITH_LOCK --> |failed - strict mode| COMMIT_FAILED
COMMIT_WITH_LOCK --> |success| COMMIT_SUCCESS
RETRY --> END
COMMIT_SUCCESS --> END
COMMIT_FAILED --> END

[BsoBird]

@JingsongLi Hi. Can you check this? Tks.

[JingsongLi]

Thanks @BsoBird ! I need time to look this, next week will give your feedback.

[JingsongLi]

Close this PR because long it is no longer active. Feel free to open a discussion in dev mail list.