[Comprehensive Issue] Fix Out-Of-Memory Issue During Bulk File Reading and Upgrade Elasticsearch Version

[wtt40122]

Description

• Overview: The current system encounters an Out-Of-Memory (OOM) issue when processing bulk file reading with wildcard matching, and we plan to upgrade Elasticsearch (ES) to the latest stable version to improve performance and security.

• Steps to Reproduce OOM Issue:

  1. Attempt to read a large number of files using wildcard pattern matching.
  2. Observe a sharp increase in system memory usage.
  3. After reaching the memory limit, the system throws an OOM error.

• Elasticsearch Version Upgrade Requirement:

  • The current ES version has known performance and security issues.
  • An upgrade to the latest stable version is needed to take advantage of new features and security updates.

• Expected Behavior:

  • The system should efficiently handle a large number of files without excessive memory consumption.
  • After upgrading, the ES version should maintain or improve performance while remaining compatible with existing data structures and queries.

• Actual Behavior:

  • The system's memory usage sharply increases when handling a large number of files, leading to an OOM error.
  • [If an ES version upgrade has been attempted, describe the issues encountered during the upgrade process]

Milestones:

• Resolve the OOM issue
• Complete the Elasticsearch version upgrade