OpenSearch Remote Store Parquet Upload and Replication Implementation

Description

This PR introduces multi-format data storage capabilities to OpenSearch, enabling the system to handle Parquet files alongside traditional Lucene segments in both local and remote storage environments. This architectural enhancement transforms OpenSearch from a single-format (Lucene-only) system to a format-agnostic storage platform.

Key Features Implemented

Format-Aware Storage Architecture: Complete redesign of storage and replication systems to support multiple data formats
Parquet Remote Storage: Full integration of Parquet files into OpenSearch's remote store functionality
Multi-Format Replication: Enhanced segment replication system that handles format-aware file transfers
Plugin-Based Format Discovery: Extensible plugin system for adding new data format support
Backward Compatibility: Maintains full compatibility with existing Lucene-based functionality

Technical Implementation

Core Architecture Changes

1. Format-Aware File Tracking

Enhanced FileMetadata: Extended to include dataFormat field enabling format identification
Multi-Format Directory Routing: New CompositeStoreDirectory routes operations to appropriate format-specific directories
Plugin-Based Format Support: DataSourcePlugin interface extended for format-specific directory and blob container creation

2. Remote Storage Multi-Format Support

CompositeRemoteDirectory: New remote storage layer with format-specific blob containers
Format-Aware Upload/Download: Complete rewrite of segment transfer mechanisms supporting multiple formats
Remote Metadata Enhancement: RemoteSegmentMetadata updated to preserve format information during serialization

3. Replication System Overhaul

ReplicationCheckpoint Redesign: Migrated from Map<String, StoreFileMetadata> to Map<FileMetadata, StoreFileMetadata> for format preservation
CatalogSnapshot Integration: Replaced SegmentInfos-based replication with CatalogSnapshot for better format abstraction
Format-Aware Diff Calculation: Enhanced segment diff algorithms to consider both filename and format

4. Storage Infrastructure Updates

Store Class Enhancement: Added CompositeStoreDirectory support with plugin-based format discovery
IndexShard Integration: Enhanced with composite engine initialization and format-aware operation support
Transfer Tracking: Updated RemoteSegmentTransferTracker to track format-specific upload statistics

Parquet-Specific Implementations

1. Directory Management

GenericStoreDirectory: New implementation for non-Lucene formats with full IndexInput/OutputStream support
Parquet Plugin Enhancement: Extended ParquetDataFormatPlugin to support FormatStoreDirectory and BlobContainer creation
Format-Specific Path Handling: Automatic path resolution for Parquet files (shard_path/parquet/)

2. Remote Operations

Parquet Blob Containers: Dedicated remote storage containers with format-specific paths
Checksum Calculation: Format-appropriate checksum algorithms (CRC32 for Parquet vs Lucene-specific methods)
Upload Optimization: Multi-stream upload support for large Parquet files with priority handling

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

[ ] Functionality includes testing.
[ ] API changes companion pull request created, if applicable.
[ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Nov 05 '25 14:11 ask-kamal-nayan

:x: Gradle check result for 7c29aeff92221737028c34c233a79063e793e8d1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 05 '25 14:11 github-actions[bot]

This looks great but I'm worried our feature branch diff is getting way too large right now, and this imo is not required for us to start pulling this back into main from the fb, can this stay on a personal fork, where our feat branch remains only primary writes?

Nov 05 '25 18:11 mch2

:x: Gradle check result for 9446a1ca995898ff2f4632e5ae7e33bada9e75b0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 07 '25 12:11 github-actions[bot]

:x: Gradle check result for 662880b7c0451f94576a0e97d6ebc56a63ca5d45: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 10 '25 19:11 github-actions[bot]

:x: Gradle check result for 3387048573c64732d6839c439a64ed80dff80d61: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 11 '25 19:11 github-actions[bot]

This looks great but I'm worried our feature branch diff is getting way too large right now, and this imo is not required for us to start pulling this back into main from the fb, can this stay on a personal fork, where our feat branch remains only primary writes?

@mch2 @ask-kamal-nayan Would it be better to create 3-4 relatively smaller PRs and keep them in sync with main fb so that there is harmony in development b/w different components and there is no major refactoring required later from storage format side?

Nov 13 '25 07:11 ashking94

:x: Gradle check result for 3385178bce28e305d3450b476dd3906cb7571de5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 16 '25 11:11 github-actions[bot]

:x: Gradle check result for 65d098a42f2ead7c38e0d7501e66e0791e67f6ea: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 17 '25 09:11 github-actions[bot]

:x: Gradle check result for b274b2045410180ca1063a2b201825923a99e13c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 19 '25 12:11 github-actions[bot]

:x: Gradle check result for 6fad8e99cf494467fb08c821a8bf0be9290159cb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Nov 19 '25 13:11 github-actions[bot]

[!IMPORTANT]

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

[ ] Create PR with unit tests
[ ] Post copyable unit tests in a comment

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Dec 11 '25 10:12 coderabbitai[bot]

:x: Gradle check result for 3f5059c50929be2f9e06b9fa1171843110348c0e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 11 '25 10:12 github-actions[bot]

:x: Gradle check result for 7c5fc3ddf004bce3b9e27fd2b67e2f93a39dd9c5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 16 '25 04:12 github-actions[bot]

:x: Gradle check result for c9d5f23ca97043b58e65c401e762b56acbc010db: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 17 '25 08:12 github-actions[bot]

:x: Gradle check result for f8a561412a50e99c47b56fc96f2ce6baacede74a: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 22 '25 07:12 github-actions[bot]

:x: Gradle check result for f0133cdb992106d483ae7c3d0c681227489371f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 22 '25 14:12 github-actions[bot]

:x: Gradle check result for de0843c7cecb8e0cc0975a7633d6fca3c172bd98: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 22 '25 22:12 github-actions[bot]

:x: Gradle check result for e4e8e12d484fb76bd4ca8e0b17e97246d231d4e0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Dec 22 '25 22:12 github-actions[bot]