OpenSearch Support ignore_above for keyword/wildcard field and optimise text field under derived source

For keyword and wildcard field handle ignore_above by explicitly storing values exceeding that and using stored field for using it as a fallback
For text field, use sub keyword fields for deriving source(using doc_values) instead of explicitly storing the value, this way we can avoid storing duplicates(as a doc_values under sub keyword and stored field under text field).
In case of multiple keyword fields, we will be going through each one to find first one, and in case of ingored field, we are choosing keyword with max value of ignore_above to save the dupplication of docValue and fallback stored field. Updation of sub keyword field is append only, we can't remove keyword once added, these would get appended in applicable keyword fields for derived source as and when mapping gets updated.

Summary by CodeRabbit

Release Notes

New Features
- Added support for ignore_above parameter on keyword and wildcard fields.
- Optimized text field retrieval in derived source scenarios.
Bug Fixes
- Improved handling of values exceeding the ignore_above threshold for keyword and wildcard fields with derived source enabled.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Nov 28 '25 03:11 tanik98

Walkthrough

This PR extends support for the ignore_above configuration to keyword and wildcard fields within derived source mappings. It introduces a composite field value fetcher mechanism to retrieve values from multiple sources (stored fields, doc values, and a dedicated ignore field) and updates core mapper classes to store values exceeding ignore_above in a special derived-source field for later retrieval.

Changes

Cohort / File(s)	Summary
Changelog `CHANGELOG.md`	Adds entry documenting support for `ignore_above` on keyword/wildcard fields and text field optimization under derived source.
Core mapper infrastructure `server/src/main/java/org/opensearch/index/mapper/CompositeFieldValueFetcher.java`, `FieldValueFetcher.java`	Introduces `CompositeFieldValueFetcher` class that chains multiple `FieldValueFetcher` instances, trying each in sequence until a non-empty result is found. Updates `FieldValueFetcher.write` to handle null value lists.
Field type additions `server/src/main/java/org/opensearch/index/mapper/StringFieldType.java`	Adds `derivedSourceIgnoreFieldName()` method and private constant `IGNORED_VALUE_FIELD_SUFFIX` to support derived source ignore field naming.
Keyword field mapper `server/src/main/java/org/opensearch/index/mapper/KeywordFieldMapper.java`	Relaxes derived-source eligibility checks for `ignore_above`; introduces `DerivedSourceHelper` to manage value fetching from primary field and fallback ignore field; stores values exceeding `ignore_above` in dedicated ignore field when derived source is enabled.
Wildcard field mapper `server/src/main/java/org/opensearch/index/mapper/WildcardFieldMapper.java`	Updates parse flow to store values exceeding `ignore_above` in derived source ignore field; relaxes guard conditions in `canDeriveSourceInternal`; replaces single `SortedSetDocValuesFetcher` with composite approach; introduces `DerivedSourceHelper`.
Text field mapper `server/src/main/java/org/opensearch/index/mapper/TextFieldMapper.java`	Extends field type construction to propagate `MultiFields`; adds public getters/setters for derived-source keyword support and ignore length tracking; injects logic to store ignore field when needed; reworks derived source field value fetching to use `CompositeFieldValueFetcher`; introduces `DerivedSourceHelper` to detect and collect keyword-derived fetchers.
Match-only text field mapper `server/src/main/java/org/opensearch/index/mapper/MatchOnlyTextFieldMapper.java`	Updates `buildFieldType` method signature to accept `MultiFields` parameter; threads multi-field configuration through field type construction.
Integration tests `server/src/internalClusterTest/java/org/opensearch/get/GetActionIT.java`, `recovery/FullRollingRestartIT.java`, `search/simple/SimpleSearchIT.java`, `update/UpdateIT.java`	Updates derived source test mappings: extends keyword sub-field definitions with `ignore_above` configuration; removes `store: true` from text fields and `doc_values: true` from wildcard fields; updates document indexing and assertions to reflect new field configurations.
Unit tests `server/src/test/java/org/opensearch/index/mapper/CompositeFieldValueFetcherTests.java`	New test class validating `CompositeFieldValueFetcher` behavior across multiple fetcher scenarios, empty lists, and exception propagation.
Keyword field mapper tests `server/src/test/java/org/opensearch/index/mapper/KeywordFieldMapperTests.java`	Adds factory method `getMapperServiceForDerivedSource` and tests for derived source behavior with `ignore_above` threshold scenarios; replaces exception expectations with no-exception assertions for canDeriveSource path.
Text field mapper tests `server/src/test/java/org/opensearch/index/mapper/TextFieldMapperTests.java`	Renames and refactors stored field test; adds multiple tests for derived source with keyword sub-fields, ignore_above thresholds, normalization, and long text handling; updates `IndexAnalyzers` construction with "whitespace" analyzer.
Wildcard field mapper tests `server/src/test/java/org/opensearch/index/mapper/WildcardFieldMapperTests.java`	Adds tests for derived value fetching and parse behavior around `ignore_above` threshold with derived source enabled; introduces `getMapperServiceForDerivedSource` helper and `assertDoesNotThrow` utility.
Object mapper tests `server/src/test/java/org/opensearch/index/mapper/ObjectMapperTests.java`	Removes multiple derive-source validation test cases (ignore_above, nested object fields, copy_to, multi-field scenarios); simplifies remaining multi-type field mapping by removing stored and doc_values flags.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Parser as Field Parser
    participant IgnoreField as Ignore Field Storage
    participant Derived as Derived Source Generator
    participant Composite as CompositeFieldValueFetcher
    participant Primary as Primary Fetcher
    participant Fallback as Fallback Fetcher

    Client->>Parser: Index document with text exceeding ignore_above
    Parser->>Parser: Check if derived source enabled & value > ignore_above
    alt Value exceeds ignore_above threshold
        Parser->>IgnoreField: Store value in derivedSourceIgnore field
        Parser->>Derived: Skip normal indexing, return early
    else Value within threshold
        Parser->>Parser: Normal field indexing
    end
    
    Note over Client,Fallback: Later: Deriving source for document
    Client->>Derived: Request derived source
    Derived->>Composite: Create fetcher chain
    Composite->>Primary: Try primary fetcher (doc values or stored)
    alt Primary has value
        Primary-->>Composite: Return value
        Composite-->>Derived: Return converted value
    else Primary empty
        Composite->>Fallback: Try fallback fetcher (ignore field)
        Fallback->>IgnoreField: Read from derivedSourceIgnore field
        IgnoreField-->>Fallback: Return stored value
        Fallback-->>Composite: Return converted value
        Composite-->>Derived: Return value
    end
    Derived-->>Client: Return derived source with retrieved value

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Areas requiring extra attention:

CompositeFieldValueFetcher logic and proper ordering of fetcher chains across multiple field mappers
Derived source storage and retrieval paths in KeywordFieldMapper, WildcardFieldMapper, and TextFieldMapper — verify null-safety and edge cases around ignore_above thresholds
Public API additions to StringFieldType and TextFieldType — confirm backward compatibility and semantic correctness of new getters/setters
DerivedSourceHelper implementations across three mapper classes — ensure consistency in keyword detection and fetcher collection logic
Multi-field wiring changes in TextFieldMapper and MatchOnlyTextFieldMapper — verify MultiFields parameter threading does not break existing functionality
Integration test updates — confirm that removed store and doc_values flags do not inadvertently change test expectations or mask regressions

Suggested labels

Search:Performance

Suggested reviewers

cwperks
gbbafna
saratvemulapalli
sachinpkale
kotwanikunal

Poem

🐰 Ignoring above, we now derive with care, Composite fetchers fetch from everywhere, Keyword and wildcard fields now play their part, Derived source shines, a clever work of art! Long text? No problem—we store what's spared! ✨

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description provides clear technical details about the changes and approach, but is missing required template sections like 'Related Issues' and the 'Check List' to confirm testing and documentation.	Add the 'Related Issues' section (should reference #20113), complete the 'Check List' by checking the appropriate boxes, and ensure all required template sections are filled out.
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.12% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly describes the main changes: adding support for ignore_above in keyword/wildcard fields and optimizing text field behavior under derived source, which aligns with the changeset.

✨ Finishing touches

[ ] 📝 Generate docstrings

🧪 Generate unit tests (beta)

[ ] Create PR with unit tests
[ ] Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}