doris icon indicating copy to clipboard operation
doris copied to clipboard

[Fix](Iam-role)Ensure StorageProperties list remains ordered when auto-loading default HDFS

Open CalvinKirs opened this issue 1 month ago • 4 comments

… When no explicit HdfsProperties is configured, the system automatically adds a default HdfsProperties instance. To prevent configuration conflicts and parameter overrides, it is essential that this default entry is always inserted at the first position of the storage list.

Previously, the ordering of the StorageProperties list was not explicitly guaranteed, which could lead to the following issues:

The auto-loaded default HDFS entry could appear after other storage providers.

Subsequent initialization steps (initNormalizeAndCheckProps, buildHadoopStorageConfig) might unintentionally override user-defined parameters.

Behavior of storage resolution became dependent on iteration order, causing subtle configuration bugs.

Previously, the catalog used:

catalogProperty.getStoragePropertiesMap().values()

to obtain the list of StorageProperties. However, getStoragePropertiesMap() stores the properties in a Map, and Map iteration order is not guaranteed. This caused the resolved storage configurations to be non-deterministic.

Problem

Some Hadoop/S3 parameters—such as: fs.s3a.aws.credentials.provider

have default values. When multiple StorageProperties instances were merged in unpredictable order, these defaults could override user-configured settings.

This created actual production issues:

When using IAM Role (Assume Role), the correct provider must be AwsAssumedRoleCredentialProvider.

Because the list order was unstable, sometimes the default provider overrode the intended Assume-Role provider.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • [ ] Regression test
    • [ ] Unit Test
    • [ ] Manual test (add detailed scripts or steps below)
    • [ ] No need to test or manual test. Explain why:
      • [ ] This is a refactor/code format and no logic has been changed.
      • [ ] Previous test can cover this change.
      • [ ] No code files have been changed.
      • [ ] Other reason
  • Behavior changed:

    • [ ] No.
    • [ ] Yes.
  • Does this need documentation?

    • [ ] No.
    • [ ] Yes.

Check List (For Reviewer who merge this PR)

  • [ ] Confirm the release note
  • [ ] Confirm test cases
  • [ ] Confirm document
  • [ ] Add branch pick label

CalvinKirs avatar Dec 11 '25 10:12 CalvinKirs

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

hello-stephen avatar Dec 11 '25 10:12 hello-stephen

run buildall

CalvinKirs avatar Dec 11 '25 10:12 CalvinKirs

FE UT Coverage Report

Increment line coverage 90.91% (10/11) :tada: Increment coverage report Complete coverage report

hello-stephen avatar Dec 11 '25 11:12 hello-stephen

FE Regression Coverage Report

Increment line coverage 100.00% (11/11) :tada: Increment coverage report Complete coverage report

hello-stephen avatar Dec 11 '25 13:12 hello-stephen

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Dec 16 '25 04:12 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Dec 16 '25 04:12 github-actions[bot]