opensearch-migrations

opensearch-migrations copied to clipboard

OpenSearch upgrades, migrations, and comparison tooling

OpenSearch upgrade, migration, and comparison tooling facilitates OpenSearch migrations and upgrades. With these tools, you can set up a proof-of-concept environment locally using Docker containers or deploy to AWS using a one-click deployment script. Once set up and deployed, users can redirect their production traffic from a source cluster to a provisioned target cluster, enabling a comparison of results between the two clusters. All traffic directed to the source cluster is stored for future replay. Meanwhile, traffic to the target cluster is replayed at an identical rate to ensure a direct "apple-to-apple" comparison. This toolset empowers users to fine-tune cluster configurations and manage workloads more effectively.

Table of Contents

• Supported cluster versions and platforms
• Build and deploy
  • Local deployment
  • AWS deployment
• Developer contributions
  • Pre-commit hooks
  • Traffic Capture Proxy and Replayer
  • Running Tests
• Security
• License

Supported cluster versions and platforms

There are numerous combinations of source clusters, target clusters, and platforms. While the tools provided in this repository might work with various combinations, they might not support breaking changes between different source and target versions. Below is a growing support table that will be updated over time.

Build and deploy

Local deployment

A containerized end-to-end solution can be deployed locally using the Docker Solution.

AWS deployment

Refer to AWS Deployment to deploy this solution to AWS.

Developer contributions

Pre-commit hooks

Developers must run the "install_githooks.sh" script in order to add any pre-commit hooks. Developers should run these hooks before opening a pull request to ensure checks pass and prevent potential rejection of the pull request."

Traffic Capture Proxy and Replayer

The TrafficCapture directory hosts a set of projects designed to facilitate the proxying and capturing of HTTP traffic, which can then be offloaded and replayed to other HTTP(S) server(s).

More documentation on this directory including the projects within it can be found here: Traffic Capture.

Fetch Migration

The FetchMigration directory hosts tools that simplify the process of backfilling / moving data from one cluster to another.

Further documentation can be found here: Fetch Migration README.

Running Tests

Developers can run a test script which will verify the end-to-end Local Docker Solution.

More documentation on this test script can be found here: End-to-End Testing

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

Releasing

The release process is standard across repositories in this org and is run by a release manager volunteering from amongst maintainers.

1. Create a tag, e.g. 0.1.0, and push it to this GitHub repository.
2. The release-drafter.yml will be automatically kicked off and a draft release will be created.
3. This draft release triggers the jenkins release workflow as a result of which the opensearch-migrations toolset is released and published on artifacts.opensearch.org example as https://artifacts.opensearch.org/migrations/0.1.0/opensearch-migrations-0.1.0.tar.gz.
4. Once the above release workflow is successful, the drafted release on GitHub is published automatically.