OpenSearch [GRPC] Add handling for the grpc.detailed_errors.enabled setting and global gRPC error

Description

This PR introduces:

a new setting grpc.detailed_errors.enabled (much like the similar http flag.)
handling of the globalParams#errorTrace request parameter

The grpc.detailed_errors.enabled has the same effect as its http counterpart http.detailed_errors.enabled:

fail fast on error_trace=true and the setting turned off
provide only a summary of an error in case error_trace=false
provide full stack trace when error_trace=true

Some of the test evidences are listed in this comment.

Related Issues

Resolves #19639

Check List

[x] Functionality includes testing.
[x] API changes companion pull request created, if applicable.
~[ ] Public documentation issue/PR created, if applicable.~

Please note, that I haven't yet created the public documentation issue/PR for this setting as first I wanted to give this PR a go, get more feedback and then create the corresponding issue/PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Summary by CodeRabbit

New Features
- Added a new server setting (grpc.detailed_errors.enabled) to control whether detailed error information is included in gRPC error responses.
- Added support for global error trace parameter handling, allowing clients to request detailed stack traces and exception information based on server configuration.
- Implemented validation to ensure consistency between client error trace requests and server settings.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Oct 15 '25 09:10 fdesu

:x: Gradle check result for 702f7d902128af216da7e108cfebdc00c8b84b6e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Oct 15 '25 10:10 github-actions[bot]

:white_check_mark: Gradle check result for 63928491975374ccac9177cadce1aba1aae401e4: SUCCESS

Oct 15 '25 12:10 github-actions[bot]

Codecov Report

:x: Patch coverage is 89.52381% with 11 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 73.28%. Comparing base (930ae74) to head (436f121). :warning: Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...ensearch/transport/grpc/util/GrpcErrorHandler.java	84.61%	6 Missing :warning:
...ailedexception/SnapshotShardFailureProtoUtils.java	60.00%	0 Missing and 2 partials :warning:
...rt/grpc/listeners/SearchRequestActionListener.java	75.00%	1 Missing :warning:
...searchexception/OpenSearchExceptionProtoUtils.java	87.50%	1 Missing :warning:
...oto/response/search/ShardStatisticsProtoUtils.java	0.00%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19644      +/-   ##
============================================
+ Coverage     73.20%   73.28%   +0.07%     
- Complexity    71745    71801      +56     
============================================
  Files          5795     5796       +1     
  Lines        328304   328341      +37     
  Branches      47283    47284       +1     
============================================
+ Hits         240334   240615     +281     
+ Misses        68663    68463     -200     
+ Partials      19307    19263      -44

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

:snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Oct 15 '25 12:10 codecov[bot]

:x: Gradle check result for 9b42790ab65b90f36873fe00f1ad6f147d8f184e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Oct 20 '25 13:10 github-actions[bot]

Tests with failures:

org.opensearch.action.admin.indices.create.CreateIndexIT.testCreateAndDeleteIndexConcurrently

org.opensearch.action.admin.indices.create.CreateIndexIT.classMethod

5600 tests completed, 2 failed, 407 skipped

Looks like a failure due to a flaky test, more here: https://github.com/opensearch-project/OpenSearch/issues/14312

Oct 21 '25 07:10 fdesu

Hi folks, this PR is ready for review, could someone please take a look? Thanks a lot!

Oct 21 '25 19:10 fdesu

Looks like a failure due to a flaky test

The CI can be rerun by closing/re-opening the PR, or pushing an empty commit. (Or by resolving conflicts which creates a merge commit)

Oct 25 '25 01:10 karenyrx

:white_check_mark: Gradle check result for bf4d92f6bd51850ea86bd6dee32fbf6f9a2b3cc2: SUCCESS

Oct 25 '25 13:10 github-actions[bot]

:x: Gradle check result for f6d8f9a958620fc0293511a19bfca84d804fe016: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 04 '25 15:11 github-actions[bot]

Side question: Do we also eventually plan for support for REST_EXCEPTION_SKIP_STACK_TRACE as well?

@karenyrx, good question. What would be use cases here and what an implementation might look like?

Nov 04 '25 19:11 fdesu

Side question: Do we also eventually plan for support for REST_EXCEPTION_SKIP_STACK_TRACE as well?

@karenyrx, good question. What would be use cases here and what an implementation might look like?

I just noticed that this exists on the REST-side, and the eventual decision on whether a stack trace should be returned or not is dependent on this global variable. Its population is also dependent on the global param error_trace though, so I suppose this can/will be handled in the next PR for https://github.com/opensearch-project/OpenSearch/issues/19092 ?

Nov 04 '25 23:11 karenyrx

:x: Gradle check result for 6f06a3ee205a35506571f16b4ba2b7626e333706: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 05 '25 14:11 github-actions[bot]

:x: Gradle check result for c4f605b5be349f7d7a9454670532dad9c487b1c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 05 '25 14:11 github-actions[bot]

:x: Gradle check result for 80b2bd20e5fb9be7cc9b3e20a0b31204fc08f95b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 05 '25 16:11 github-actions[bot]

:x: Gradle check result for b4f51b035fae1de090fb43a16ccc6a31bd5a9f57: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 05 '25 21:11 github-actions[bot]

:x: Gradle check result for 0c3696fbff304669bfad4f96df0c17236f4ca8df: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 06 '25 09:11 github-actions[bot]

:white_check_mark: Gradle check result for 3f5228164e36f8dba109529d8ceaaeb45c1b13c2: SUCCESS

Nov 06 '25 14:11 github-actions[bot]

Here are a few examples to better show the outcome of this change.

Http Bulk endpoint (both error_trace=true and false produce the same outcome):

Http Bulk

{
    "took": 2,
    "errors": true,
    "items": [
        {
            "update": {
                "_index": "movies",
                "_id": "9999999999999999999999999999999999",
                "status": 404,
                "error": {
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "index": "movies",
                    "shard": "0",
                    "index_uuid": "IlilUpNfTTyqJBAD_rZRFQ"
                }
            }
        }
    ]
}

Grpc Before:

Grpc Bulk

{
    "items": [
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "sCA04AioTuGjeOlMDf8J7g"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "stack_trace": "[movies/sCA04AioTuGjeOlMDf8J7g][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:602)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        }
    ],
    "errors": true,
    "took": "1"
}

Grpc After:

Grpc Bulk

error_trace=false

{
    "items": [
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "sCA04AioTuGjeOlMDf8J7g"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing"
                }
            }
        }
    ],
    "errors": true,
    "took": "243"
}

error_trace=true

{
    "items": [
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "sCA04AioTuGjeOlMDf8J7g"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "stack_trace": "[movies/sCA04AioTuGjeOlMDf8J7g][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:602)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        }
    ],
    "errors": true,
    "took": "1"
}

Note that Http Bulk, if itself failed unexpectedly with e.g. 500 (not one of the bulk operations), would respect the error_trace

Grpc Bulk fails with 500 with error_trace=true

{
    "error": {
        "root_cause": [
            {
                "type": "null_pointer_exception",
                "reason": "Cannot store to byte/boolean array because \"this._outputBuffer\" is null",
                "stack_trace": "OpenSearchException[Cannot store to byte/boolean array because \"this._outputBuffer\" is null]; nested: NullPointerException[Cannot store to byte/boolean array because \"this._outputBuffer\" is null];\n\tat org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:710)\n\tat org.opensearch.OpenSearchException.generateFailureXContent(OpenSearchException.java:637)\n\tat org.opensearch.rest.BytesRestResponse.build(BytesRestResponse.java:172)\n\tat org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:132)\n\tat org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:109)\n\tat org.opensearch.rest.action.RestActionListener.from(RestActionListener.java:74)\n\tat org.opensearch.rest.action.RestActionListener.onFailure(RestActionListener.java:88)\n\tat org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:66)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:768)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:741)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:727)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1253)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1161)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1147)\n\tat org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1815)\n\tat org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1587)\n\tat org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1680)\n\tat org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1660)\n\tat org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72)\n\tat org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62)\n\tat org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$0(TransportReplicationAction.java:638)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)\n\tat org.opensearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:431)\n\tat org.opensearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:417)\n\tat org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:186)\n\tat org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:178)\n\tat org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:372)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:518)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$1(TransportWriteAction.java:547)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:145)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:123)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97)\n\tat org.opensearch.index.shard.IndexShard.sync(IndexShard.java:4782)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:545)\n\tat org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:379)\n\tat org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:178)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)\n\tat org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:355)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:553)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:496)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\nCaused by: java.lang.NullPointerException: Cannot store to byte/boolean array because \"this._outputBuffer\" is null\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeStartObject(UTF8JsonGenerator.java:387)\n\tat org.opensearch.common.xcontent.json.JsonXContentGenerator.writeStartObject(JsonXContentGenerator.java:168)\n\tat org.opensearch.core.xcontent.XContentBuilder.startObject(XContentBuilder.java:289)\n\tat org.opensearch.action.bulk.BulkResponse.toXContent(BulkResponse.java:172)\n\tat org.opensearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:72)\n\tat org.opensearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:48)\n\tat org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:57)\n\tat org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:49)\n\tat org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52)\n\tat org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64)\n\t... 52 more\n"
            }
        ],
        "type": "null_pointer_exception",
        "reason": "Cannot store to byte/boolean array because \"this._outputBuffer\" is null",
        "stack_trace": "java.lang.NullPointerException: Cannot store to byte/boolean array because \"this._outputBuffer\" is null\n\tat com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeStartObject(UTF8JsonGenerator.java:387)\n\tat org.opensearch.common.xcontent.json.JsonXContentGenerator.writeStartObject(JsonXContentGenerator.java:168)\n\tat org.opensearch.core.xcontent.XContentBuilder.startObject(XContentBuilder.java:289)\n\tat org.opensearch.action.bulk.BulkResponse.toXContent(BulkResponse.java:172)\n\tat org.opensearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:72)\n\tat org.opensearch.rest.action.RestStatusToXContentListener.buildResponse(RestStatusToXContentListener.java:48)\n\tat org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:57)\n\tat org.opensearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:49)\n\tat org.opensearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:52)\n\tat org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:64)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:768)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:741)\n\tat org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:727)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)\n\tat org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1253)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1161)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1147)\n\tat org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1815)\n\tat org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1587)\n\tat org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1680)\n\tat org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1660)\n\tat org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72)\n\tat org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62)\n\tat org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45)\n\tat org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)\n\tat org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$0(TransportReplicationAction.java:638)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)\n\tat org.opensearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:431)\n\tat org.opensearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:417)\n\tat org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:186)\n\tat org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:178)\n\tat org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:372)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:518)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$1(TransportWriteAction.java:547)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:145)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:123)\n\tat org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97)\n\tat org.opensearch.index.shard.IndexShard.sync(IndexShard.java:4782)\n\tat org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:545)\n\tat org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:379)\n\tat org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:178)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)\n\tat org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:355)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:553)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:496)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
    },
    "status": 500
}

Also, here's an example of Http Update Endpoint

Http Update Endpoint Examples

error_trace=false

{
    "error": {
        "root_cause": [
            {
                "type": "document_missing_exception",
                "reason": "[9999999999999999999999999999999999]: document missing",
                "index": "movies",
                "shard": "0",
                "index_uuid": "sCA04AioTuGjeOlMDf8J7g"
            }
        ],
        "type": "document_missing_exception",
        "reason": "[9999999999999999999999999999999999]: document missing",
        "index": "movies",
        "shard": "0",
        "index_uuid": "sCA04AioTuGjeOlMDf8J7g"
    },
    "status": 404
}

error_trace=true

{
    "error": {
        "root_cause": [
            {
                "type": "document_missing_exception",
                "reason": "[9999999999999999999999999999999999]: document missing",
                "index": "movies",
                "shard": "0",
                "index_uuid": "lg0Ji3cYQzybfTolA9rGQw",
                "stack_trace": "[movies/lg0Ji3cYQzybfTolA9rGQw][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:236)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:229)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:95)\n\tat org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler$1.doRun(TransportInstanceSingleOperationAction.java:338)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
            }
        ],
        "type": "document_missing_exception",
        "reason": "[9999999999999999999999999999999999]: document missing",
        "index": "movies",
        "shard": "0",
        "index_uuid": "lg0Ji3cYQzybfTolA9rGQw",
        "stack_trace": "[movies/lg0Ji3cYQzybfTolA9rGQw][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:236)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:229)\n\tat org.opensearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:95)\n\tat org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler$1.doRun(TransportInstanceSingleOperationAction.java:338)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
    },
    "status": 404
}

A few details on the setting being disabled scenario

Setting disabled HTTP...

error_trace=false

{
    "took": 1,
    "errors": true,
    "items": [
        {
            "update": {
                "_index": "movies",
                "_id": "9999999999999999999999999999999999",
                "status": 404,
                "error": {
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "index": "movies",
                    "shard": "0",
                    "index_uuid": "ua7x-Q2QSIalHMuMOVV-Ew"
                }
            }
        }
    ]
}

error_trace=true

{
    "error": "error traces in responses are disabled.",
    "status": 400
}

...and the Grpc side

Setting disabled Grpc...

error_trace=false

{
    "items": [
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "ua7x-Q2QSIalHMuMOVV-Ew"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing"
                }
            }
        }
    ],
    "errors": true,
    "took": "2"
}

error_trace=true

Status Code 3, Invalid Argument

see more on the status code here

Also, worth mentioning that we currently don't populate items.operation.error.root_cause (as e.g. in the Http Update Endpoint examples) which we can take a look at separately.

Nov 06 '25 15:11 fdesu

Http Bulk endpoint (both error_trace=true and false produce the same outcome):

Is this a bug in HTTP bulk (should error_trace=true not pritn out the stack trace)? Or is this intentional and the gRPC side also be following the same mechanism to always have error_trace=true and false both produce the same outcome?

Note that Http Bulk, if itself failed unexpectedly with e.g. 500 (not one of the bulk operations), would respect the error_trace

Can you confirm, does GRPC also do this? There was no example of GRPC with 500 code with error_trace=false

Status Code 3, Invalid Argument

Was there any metadata printed alongside this, outlining the reason for it? If not, we should have that as well (separate issue can be created for this if so)

Also, here's an example of Http Update Endpoint [...] Also, worth mentioning that we currently don't populate items.operation.error.root_cause (as e.g. in the Http Update Endpoint examples) which we can take a look at separately.

(GRPC does not have an UpdaterDocument endpoint, so I assume you mean the Bulk update operation here.) Could you share the GRPC response for the Bulk update error response, to compare with HTTP? Does it only happen for update, or also for other uoperations? Agree, I think if this is happening, we can file a separate issue for it as well.

Lastly, besides bulk, do we have any search response examples?

Nov 06 '25 19:11 karenyrx

@karenyrx

Is this a bug in HTTP bulk (should error_trace=true not pritn out the stack trace)? Or is this intentional and the gRPC side also be following the same mechanism to always have error_trace=true and false both produce the same outcome?

Great question, I wanted to ask around and file a new issue later if it's indeed a bug. It looks like a tiny miss, as the OpenSearchException.REST_EXCEPTION_SKIP_STACK_TRACE = "rest.exception.stacktrace.skip" is not being properly propagated to the org.opensearch.rest.action.RestStatusToXContentListener#buildResponse while building a response containing an array of BulkResponseItem with, potentially, errors.

This could also be a deliberate decision as the response with errors with detailed stacktraces might be much bigger than the inbound request (in case of many failures). So, I'd definitely look further but want to take it separately if it's ok.

Was there any metadata printed alongside this, outlining the reason for it? If not, we should have that as well (separate issue can be created for this if so)

Definitely, here's the verbose output for the failing call:

grpcurl output

Response headers received:
(empty)

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: InvalidArgument
  Message: java.lang.IllegalArgumentException: error traces in responses are disabled.
	at org.opensearch.transport.grpc.util.GrpcParamsHandler.validateStackTraceDetailsConfiguration(GrpcParamsHandler.java:60)
	at org.opensearch.transport.grpc.services.DocumentServiceImpl.bulk(DocumentServiceImpl.java:48)
	at org.opensearch.protobufs.services.DocumentServiceGrpc$MethodHandlers.invoke(DocumentServiceGrpc.java:505)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:356)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:861)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)

(GRPC does not have an UpdaterDocument endpoint, so I assume you mean the Bulk update operation here.) Could you share the GRPC response for the Bulk update error response, to compare with HTTP? Does it only happen for update, or also for other uoperations? Agree, I think if this is happening, we can file a separate issue for it as well.

Some more examples for Bulk operations

{
    "items": [
        {
            "delete": {
                "x_index": "movies",
                "status": 10,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "cQhVcRjrSW-ev72Zo46npA"
                        }
                    },
                    "header": {},
                    "type": "version_conflict_engine_exception",
                    "reason": "[9999999999999999999999999999999999]: version conflict, required seqNo [12345], primary term [12345]. but no document was found",
                    "stack_trace": "[movies/cQhVcRjrSW-ev72Zo46npA][[movies][0]] VersionConflictEngineException[[9999999999999999999999999999999999]: version conflict, required seqNo [12345], primary term [12345]. but no document was found]\n"
                }
            }
        },
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "cQhVcRjrSW-ev72Zo46npA"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "stack_trace": "[movies/cQhVcRjrSW-ev72Zo46npA][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:602)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        },
        {
            "create": {
                "x_index": "movies",
                "status": 3,
                "x_id": {
                    "string": "8888888888888888888888888888888888"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {},
                    "header": {},
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse, document is empty",
                    "stack_trace": "MapperParsingException[failed to parse, document is empty]\n\tat org.opensearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:203)\n\tat org.opensearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:99)\n\tat org.opensearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:253)\n\tat org.opensearch.index.engine.Engine.prepareIndex(Engine.java:1635)\n\tat org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1207)\n\tat org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:1124)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:655)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        }
    ],
    "errors": true,
    "took": "3"
}

Lastly, besides bulk, do we have any search response examples?

Yeah, here's a few examples:

Http

error_trace=false

{
    "error": {
        "root_cause": [
            {
                "type": "index_not_found_exception",
                "reason": "no such index [qwerty]",
                "index": "qwerty",
                "resource.id": "qwerty",
                "resource.type": "index_or_alias",
                "index_uuid": "_na_"
            }
        ],
        "type": "index_not_found_exception",
        "reason": "no such index [qwerty]",
        "index": "qwerty",
        "resource.id": "qwerty",
        "resource.type": "index_or_alias",
        "index_uuid": "_na_"
    },
    "status": 404
}

error_trace=true

{
    "error": {
        "root_cause": [
            {
                "type": "index_not_found_exception",
                "reason": "no such index [qwerty]",
                "index": "qwerty",
                "resource.id": "qwerty",
                "resource.type": "index_or_alias",
                "index_uuid": "_na_",
                "stack_trace": "[qwerty] IndexNotFoundException[no such index [qwerty]]\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1212)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1149)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1099)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:332)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:292)\n\tat org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:1033)\n\tat org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073)\n\tat org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:881)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:517)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:486)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73)\n\tat org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:493)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:330)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:140)\n\tat org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)\n\tat org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113)\n\tat org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:106)\n\tat org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501)\n\tat org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$3(RestSearchAction.java:170)\n\tat org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:132)\n\tat org.opensearch.rest.RestController.dispatchRequest(RestController.java:381)\n\tat org.opensearch.rest.RestController.tryAllHandlers(RestController.java:467)\n\tat org.opensearch.rest.RestController.dispatchRequest(RestController.java:287)\n\tat org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:374)\n\tat org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:482)\n\tat org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357)\n\tat org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56)\n\tat org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42)\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:166)\n\tat io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:48)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
            }
        ],
        "type": "index_not_found_exception",
        "reason": "no such index [qwerty]",
        "index": "qwerty",
        "resource.id": "qwerty",
        "resource.type": "index_or_alias",
        "index_uuid": "_na_",
        "stack_trace": "[qwerty] IndexNotFoundException[no such index [qwerty]]\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1212)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1149)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1099)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:332)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:292)\n\tat org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:1033)\n\tat org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073)\n\tat org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:881)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:517)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:486)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73)\n\tat org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:493)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:330)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:140)\n\tat org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)\n\tat org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113)\n\tat org.opensearch.rest.action.RestCancellableNodeClient.doExecute(RestCancellableNodeClient.java:106)\n\tat org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501)\n\tat org.opensearch.rest.action.search.RestSearchAction.lambda$prepareRequest$3(RestSearchAction.java:170)\n\tat org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:132)\n\tat org.opensearch.rest.RestController.dispatchRequest(RestController.java:381)\n\tat org.opensearch.rest.RestController.tryAllHandlers(RestController.java:467)\n\tat org.opensearch.rest.RestController.dispatchRequest(RestController.java:287)\n\tat org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:374)\n\tat org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:482)\n\tat org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357)\n\tat org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56)\n\tat org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42)\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:120)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:166)\n\tat io.netty.handler.codec.http.HttpContentDecoder.decode(HttpContentDecoder.java:48)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:91)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
    },
    "status": 404
}

vs

Grpc

error_trace=false

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: NotFound
  Message: IndexNotFoundException[no such index [qwerty]]; details={"error":"IndexNotFoundException[no such index [qwerty]]","status":404}

error_trace=true

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: NotFound
  Message: IndexNotFoundException[no such index [qwerty]]; details={"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}

-> the error_trace=true seem to completely lack the stacktrace as compared to Http response. I'll look into it!

Nov 06 '25 21:11 fdesu

@karenyrx After the latest changes here's how the Grpc errors look like:

Search Endpoint

error_trace=false

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: NotFound
  Message: IndexNotFoundException[no such index [qwerty]]; details={"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}
➜  ~ grpcurl -vv -plaintext -d '{"index": ["qwerty"], "global_params": { "error_trace": true}, "q": "abc:def"}' localhost:9400 org.opensearch.protobufs.services.SearchService/Search

error_trace=true

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: NotFound
  Message: IndexNotFoundException[no such index [qwerty]]; details={"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_","stack_trace":"[qwerty] IndexNotFoundException[no such index [qwerty]]\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1212)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1149)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1099)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:332)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:292)\n\tat org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:1033)\n\tat org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073)\n\tat org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:881)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:517)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:486)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73)\n\tat org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:493)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:330)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:140)\n\tat org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)\n\tat org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113)\n\tat org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100)\n\tat org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501)\n\tat org.opensearch.transport.client.support.AbstractClient.search(AbstractClient.java:632)\n\tat org.opensearch.transport.grpc.services.SearchServiceImpl.search(SearchServiceImpl.java:69)\n\tat org.opensearch.protobufs.services.SearchServiceGrpc$MethodHandlers.invoke(SearchServiceGrpc.java:265)\n\tat io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)\n\tat io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:356)\n\tat io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:861)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"}],"type":"index_not_found_exception","reason":"no such index [qwerty]","index":"qwerty","resource.id":"qwerty","resource.type":"index_or_alias","index_uuid":"_na_","stack_trace":"[qwerty] IndexNotFoundException[no such index [qwerty]]\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1212)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1149)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1099)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:332)\n\tat org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteResolvedIndices(IndexNameExpressionResolver.java:292)\n\tat org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:1033)\n\tat org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073)\n\tat org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:881)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:517)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140)\n\tat org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105)\n\tat org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:486)\n\tat org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)\n\tat org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73)\n\tat org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:493)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:330)\n\tat org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:140)\n\tat org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)\n\tat org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)\n\tat org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113)\n\tat org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100)\n\tat org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501)\n\tat org.opensearch.transport.client.support.AbstractClient.search(AbstractClient.java:632)\n\tat org.opensearch.transport.grpc.services.SearchServiceImpl.search(SearchServiceImpl.java:69)\n\tat org.opensearch.protobufs.services.SearchServiceGrpc$MethodHandlers.invoke(SearchServiceGrpc.java:265)\n\tat io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)\n\tat io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:356)\n\tat io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:861)\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\n\tat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"},"status":404}

...and the Bulk API is still OK

error_trace=false

{
    "items": [
        {
            "delete": {
                "x_index": "movies",
                "status": 10,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "2UTclJhDT42BA8Zr3EkhqQ"
                        }
                    },
                    "header": {},
                    "type": "version_conflict_engine_exception",
                    "reason": "[9999999999999999999999999999999999]: version conflict, required seqNo [12345], primary term [12345]. but no document was found"
                }
            }
        },
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "2UTclJhDT42BA8Zr3EkhqQ"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing"
                }
            }
        },
        {
            "create": {
                "x_index": "movies",
                "status": 3,
                "x_id": {
                    "string": "8888888888888888888888888888888888"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {},
                    "header": {},
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse, document is empty"
                }
            }
        }
    ],
    "errors": true,
    "took": "3"
}

error_trace=true

{
    "items": [
        {
            "delete": {
                "x_index": "movies",
                "status": 10,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "2UTclJhDT42BA8Zr3EkhqQ"
                        }
                    },
                    "header": {},
                    "type": "version_conflict_engine_exception",
                    "reason": "[9999999999999999999999999999999999]: version conflict, required seqNo [12345], primary term [12345]. but no document was found",
                    "stack_trace": "[movies/2UTclJhDT42BA8Zr3EkhqQ][[movies][0]] VersionConflictEngineException[[9999999999999999999999999999999999]: version conflict, required seqNo [12345], primary term [12345]. but no document was found]\n"
                }
            }
        },
        {
            "update": {
                "x_index": "movies",
                "status": 5,
                "x_id": {
                    "string": "9999999999999999999999999999999999"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {
                        "index": {
                            "string": "movies"
                        },
                        "shard": {
                            "string": "0"
                        },
                        "index_uuid": {
                            "string": "2UTclJhDT42BA8Zr3EkhqQ"
                        }
                    },
                    "header": {},
                    "type": "document_missing_exception",
                    "reason": "[9999999999999999999999999999999999]: document missing",
                    "stack_trace": "[movies/2UTclJhDT42BA8Zr3EkhqQ][[movies][0]] DocumentMissingException[[9999999999999999999999999999999999]: document missing]\n\tat org.opensearch.action.update.UpdateHelper.prepareUpsert(UpdateHelper.java:149)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:101)\n\tat org.opensearch.action.update.UpdateHelper.prepare(UpdateHelper.java:91)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:602)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        },
        {
            "create": {
                "x_index": "movies",
                "status": 3,
                "x_id": {
                    "string": "8888888888888888888888888888888888"
                },
                "error": {
                    "root_cause": [],
                    "suppressed": [],
                    "metadata": {},
                    "header": {},
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse, document is empty",
                    "stack_trace": "MapperParsingException[failed to parse, document is empty]\n\tat org.opensearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:203)\n\tat org.opensearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:99)\n\tat org.opensearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:253)\n\tat org.opensearch.index.engine.Engine.prepareIndex(Engine.java:1635)\n\tat org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1207)\n\tat org.opensearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:1124)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:655)\n\tat org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:481)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)\n\tat org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)\n\tat org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)\n\tat java.base/java.lang.Thread.run(Thread.java:1474)\n"
                }
            }
        }
    ],
    "errors": true,
    "took": "2"
}

Note that Http Bulk, if itself failed unexpectedly with e.g. 500 (not one of the bulk operations), would respect the error_trace

Can you confirm, does GRPC also do this? There was no example of GRPC with 500 code with error_trace=false

Absolutely! A bit artificial example, but hope it works:

Internal error

error_trace=false

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: Internal
  Message: Something went wrong here!

error_trace=true

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: Internal
  Message: java.lang.NullPointerException: Something went wrong here!
	at org.opensearch.transport.grpc.proto.response.document.bulk.BulkResponseProtoUtils.toProto(BulkResponseProtoUtils.java:78)
	at org.opensearch.transport.grpc.listeners.BulkRequestActionListener.onResponse(BulkRequestActionListener.java:54)
	at org.opensearch.transport.grpc.listeners.BulkRequestActionListener.onResponse(BulkRequestActionListener.java:27)
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
	at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)
	at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:766)
	at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:739)
	at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:725)
	at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
	at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1253)
	at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1161)
	at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1147)
	at org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1815)
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1587)
	at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1680)
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1660)
	at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72)
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62)
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45)
	at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301)
	at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$0(TransportReplicationAction.java:638)
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
	at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)
	at org.opensearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:431)
	at org.opensearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:417)
	at org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:186)
	at org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:178)
	at org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:372)
	at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:518)
	at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:532)
	at org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:379)
	at org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:178)
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
	at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182)
	at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:355)
	at org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:553)
	at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:496)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:565)
	at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:426)
	at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:128)
	at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975)
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)

Nov 07 '25 11:11 fdesu

:white_check_mark: Gradle check result for e2b0a5ff6cf3f82fc024d17b02d72140a5a0483a: SUCCESS

Nov 07 '25 12:11 github-actions[bot]

Great question, I wanted to ask around and file a new issue later if it's indeed a bug. It looks like a tiny miss, as the OpenSearchException.REST_EXCEPTION_SKIP_STACK_TRACE = "rest.exception.stacktrace.skip" is not being properly propagated to the org.opensearch.rest.action.RestStatusToXContentListener#buildResponse while building a response containing an array of BulkResponseItem with, potentially, errors.

This could also be a deliberate decision as the response with errors with detailed stacktraces might be much bigger than the inbound request (in case of many failures). So, I'd definitely look further but want to take it separately if it's ok.

Yes, taking it separately makes sense. Would you add mind creating an issue for it?

-> the error_trace=true seem to completely lack the stacktrace as compared to Http response. I'll look into it!

Thanks! This would be great to be added to the PR.

Nov 07 '25 16:11 karenyrx

:grey_exclamation: Gradle check result for 9cd05755113a54ef08ce441680efa678eb19dbab: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Nov 10 '25 18:11 github-actions[bot]

@karenyrx could you please have a quick look? All the points seem to be covered and as for the Http Bullk API handling of error_trace, I've raised a separate issue https://github.com/opensearch-project/OpenSearch/issues/19945. Happy to polish things further if required!

Nov 11 '25 19:11 fdesu

:x: Gradle check result for 7863268c42b73ec1a15f6a0dcb4a63470176b411: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 14 '25 13:11 github-actions[bot]

:x: Gradle check result for 06adbf2c9425aaf7998c096fab0996dac819b6ea: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 14 '25 14:11 github-actions[bot]

:x: Gradle check result for ba7fae35f1872a0e98f2fc13ab83e6eb7a5a8761: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 14 '25 17:11 github-actions[bot]

:white_check_mark: Gradle check result for 0737559c64496ef61dae69d4ac33d20903a58c22: SUCCESS

Nov 14 '25 19:11 github-actions[bot]

:white_check_mark: Gradle check result for 8767699e685f07573ae181e4bcb24d4032e9752f: SUCCESS

Nov 17 '25 18:11 github-actions[bot]

[GRPC] Add handling for the grpc.detailed_errors.enabled setting and global gRPC error_trace parameter

Description

Related Issues

Check List

Summary by CodeRabbit

Codecov Report