OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

[BUG] S3.client settings missing from list of repository-s3 settings

Open aphilippejolivel opened this issue 1 month ago • 2 comments

Describe the bug

I have a problem right now using Oracled Bucket with the remote storage

Exception occurred while uploading remote index paths for 
   indexes=[top_queries-2025.11.30-85662/OlMiRO3eQDmMREEf0_gBQg],[.ql-datasou....e3-15363/hCmCFo-gS22nIaqfCUnpnA]
     
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception:
   software.amazon.awssdk.services.s3.model.S3Exception: AWS chunked encoding not supported. (Service: S3, Status Code: 501, 
   Request ID: fra-1:qd07EryGbB2TqNOyZ_YhB5w9EK5Kz34JZtDo_U12-cEFjiJFUr6x67MWO8DUoSrB) (SDK Attempt Count: 1)
     
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: AWS chunked encoding not
   supported. (Service: S3, Status Code: 501, Request ID: 
   fra-1:qd07EryGbB2TqNOyZ_YhB5w9EK5Kz34JZtDo_U12-cEFjiJFUr6x67MWO8DUoSrB) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113) ~[?:
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61) ~[?:

So I tried to disable the encoding as per the setting here in my opensearch.yml (helm chart)

https://github.com/opensearch-project/OpenSearch/blob/b7f013f27f18fea78b1aecd90dfbd58ecbfde722/plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java#L246

but got an error

SettingsException[unknown setting [s3.client.default.disable_chunked_encoding] please check that any required plugins are installed, or check the breaking changes documentation for removed settings]

it seems similar to the problem fixed by

https://github.com/opensearch-project/OpenSearch/pull/19788

Is it possible to expose all settings available in the client from the opensearch.yml file as tehre is no other way to modify them with the helm chart ?

Related component

Plugins

To Reproduce

  1. add s3.client.default.disable_chunked_encoding: true to your opensearch.yml

Expected behavior

Clean start of the app

Additional Details

Plugins repository-s3

Host/Environment (please complete the following information):

  • Kubernetes 1.34.1
  • Helm chart version 3.3.2

aphilippejolivel avatar Dec 03 '25 14:12 aphilippejolivel

Hi , I’d like to work on this issue as a first contribution.

What I understand so far: The setting s3.client..disable_chunked_encoding is defined in S3ClientSettings.java but isn’t added to the plugin’s accepted client-settings list. That’s why adding it in opensearch.yml causes unknown setting.

What I plan to do: Add DISABLE_CHUNKED_ENCODING into the CLIENT_SETTINGS list so it becomes a valid client-setting. Update documentation to reflect that this can be configured via opensearch.yml .

If this approach looks good to maintainers/owners of the repository-s3 plugin, I will submit a PR soon. Thanks!

Gautam-aman avatar Dec 03 '25 17:12 Gautam-aman

Imho it would maybe benefit from a larger fix and expose all the settings currently missing ( unless there is a problem with modifying one of them )

  • REQUEST_TIMEOUT_SETTING
  • CONNECTION_TIMEOUT_SETTING
  • CONNECTION_TTL_SETTING
  • MAX_CONNECTIONS_SETTING
  • MAX_SYNC_CONNECTIONS_SETTING
  • CONNECTION_ACQUISITION_TIMEOUT
  • MAX_PENDING_CONNECTION_ACQUIRES
  • DISABLE_CHUNKED_ENCODING

aphilippejolivel avatar Dec 04 '25 13:12 aphilippejolivel