Bump `elasticsearch-java` to version 9.x

[ppkarwasz]

Description

Since elasticsearch-java version 9.x has breaking changes, it can not be upgraded automatically.

[ramanathan1504]

Proposal: Upgrade elasticsearch-java Dependency to 9.x in Log4j2

Background

The current Log4j2 integration with Elasticsearch relies on elasticsearch-java version 8.x. With the release of version 9.x, the Elasticsearch Java client introduces significant API changes and enhancements that require explicit handling in the Log4j2 Elasticsearch appender.

Motivation

• Compatibility: Ensure Log4j2 remains compatible with modern Elasticsearch clusters that require or benefit from the 9.x client.
• Modern Features: Leverage improvements in performance, security, and the more modern, maintained APIs available in the new client.
• Reduce Technical Debt: Proactively adapt to new versions of core dependencies to avoid the challenges of relying on outdated and potentially unsupported client versions.

Challenges

• Breaking Changes: The 9.x client is not backward-compatible with the 8.x client. This includes changes to API contracts, method signatures, and builder patterns.
• Backward Compatibility: A key challenge is to manage the transition without breaking existing setups for users who are still on older versions of the Elasticsearch client.
• Testing Overhead: Additional integration testing will be required to validate that Log4j2 appenders function correctly with both old and new client versions against corresponding Elasticsearch clusters.

Proposed Approach

1. Analysis Phase

• Identify Breaking Changes: Conduct a thorough analysis of the differences between elasticsearch-java 8.x and 9.x, focusing on areas directly impacting the Log4j2 appender (e.g., request/response handling, transport layer changes, authentication).
• Map API Usage: Map the current implementation of the Elasticsearch appenders to the new APIs and patterns introduced in the 9.x client.

2. Implementation Options

• Option A: Direct Upgrade

  • Update the Log4j2 appenders to fully support the 9.x client.
  • This would mean dropping support for the 8.x client entirely in a future major release of Log4j2.
  • Pros: Simpler implementation.
  • Cons: Disruptive for users unable to upgrade their Elasticsearch client immediately.

• Option B: Dual Support Layer (Recommended)

  • Introduce an abstraction layer over the Elasticsearch client interactions within Log4j2.
  • Create two separate implementation modules or adapters: one for the 8.x client and one for the 9.x client.
  • The correct implementation would be chosen at runtime based on the elasticsearch-java version available on the classpath.
  • Pros: Ensures a smoother migration path for all users and maintains backward compatibility.
  • Cons: More complex to implement and maintain initially.

3. Testing Strategy

• Expand Integration Tests: Add new integration test suites that run against both Elasticsearch 8.x and 9.x clusters.
• Validate Core Functionality: Verify that logging events are correctly created, sent, and indexed in both scenarios.
• Test Edge Cases: Validate error handling, bulk request processing, and various authentication mechanisms (e.g., API Key, Basic Auth) for both client versions.

4. Documentation Updates

• Update Log4j2 Manual: Clearly document which versions of the elasticsearch-java client are supported.
• Provide Migration Guide: Create clear migration notes for users, explaining how to configure the appender when upgrading from elasticsearch-java 8.x to 9.x. This should include dependency changes and any necessary configuration adjustments.

[vy]

@ramanathan1504, ES is only used for tests. You can go ahead and upgrade to 9.x, and implement necessary fixes. No doc updates are necessary.

As a personal note, I'd appreciate genuine human-written to-the-point communication rather than sharing AI-provided responses verbatim when the issue description is used as a prompt.

[ramanathan1504]

ok I will start the process as per above..

I acknowledge here after I will make my own written

thanks @vy