Add a plugin for hardware-accelerated compression.

[mulugetam]

Description

This plugin adds hardware-accelerated (de)compression for the DEFLATE and LZ4 compression algorithms. The hardware that is targeted here is Intel QAT, which is now an integrated built-in accelerator on the latest 4th and 5th Gen Intel Xeon processors.

OpenSearch use can be enabled by setting two configuration parameters:

index.codec: 'QDEFLATE'
index.codec.mode: 'hardware' 

QLZ4 may also be used for index.codec. In cases where a software-failover is required, index.codec.mode must be set to auto.

JMH benchmark results, measuring in-memory (de)compression performance, show substantial gains for QDEFLATE and QLZ4 against their corresponding software implementations. We also measured significant gains for OpenSearch indexing and search workloads. Snapshot compression, though not addressed in this PR, is also a good candidate that would benefit from hardware-accelerated compression.

The plugin uses the qat-java library.

Check List

• [X] New functionality includes testing.
  • [X] All tests pass
• [X] New functionality has been documented.
  • [X] New functionality has javadoc added
• [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• [X] Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[reta]

@mulugetam please target the pull request to this repository [1], thank you

[1] https://github.com/OpenSearch-project/custom-codecs

[github-actions[bot]]

:x: Gradle check result for 5767aeff632637edcbd2e023c29152526871089a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Compatibility status:

Checks if related components are compatible with change 5767aef

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

[sarthakaggarwal97]

+1 to @reta's suggestion. @mulugetam It would be great if you can share the performance gains as well.

[mulugetam]

Closing this. New PR: https://github.com/opensearch-project/custom-codecs/pull/122.