redis-connect-dist

redis-connect-dist copied to clipboard

Redis Connect :rocket: is a distributed platform that enables real-time event streaming, transformation, and propagation of changed-data events from heterogeneous data platforms to Azure Cache for Redis, Redis Cloud, and Redis Enterprise.

:white_check_mark: No Code :white_check_mark: Seamless Integration :white_check_mark: Multi-Tenancy :white_check_mark: Linear-Scalability :white_check_mark: High-Availability :white_check_mark: Support

End-to-End demos

Table of Contents

• Background
• Quick start
• Requirements

Background

Cloud-Native Architecture Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.
	Multi-Tenancy | Partitioning | Linear Scalability Redis Connect manages multi-tenant replication pipelines. A pipeline from source to sink is known as a job. Jobs with different source types can be collocated without becoming noisy neighbors. Jobs can be partitioned for linear scalability across one or more cluster nodes.
High-Availability | Recovery Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the event of a node failure or network split, in-flight jobs will fail over to another node and seamlessly begin replication from the last committed checkpoint. Data is not lost, and order is preserved. Redis Connect works in container orchestration environments such as Kubernetes and OpenShift.
	Custom Transformations Redis Connect jobs support user-defined business logic. You can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. You can also extend the supported list of target sinks.
REST API | CLI | Swagger UI Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. You can configure, start, stop, migrate, and restart jobs using the built-in REST API and interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.
	Enterprise-Grade Security Redis Connect jobs are stateless, so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s security, including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored in Redis Connect; these secrets can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.

Requirements

Minimum production hardware requirements

• 1 GB of RAM
• 4 CPU cores
• 20 GB of disk space
• 1 Gbps network

Runtime requirements

• JRE 11+ (JRE 17+ version 0.10.7 onwards) e.g. Azul OpenJDK

Quick Start

You can run Redis Connect as a container or by downloading the code and running in your environment of choice.

Docker

You can run and deploy Redis Connect using the Redis Connect Docker image.

Download

Download the latest release for Linux or Windows and unarchive redis-connect-<version>.<build>.[tar.gz|zip] archive

The following subdirectories will be extracted under /redis-connect:

• bin – Startup scripts
• lib – Dependencies
• config – Credentials property files, jobmanager.properties, and job-config (JSON) examples
• extlib – Custom/external dependencies (e.g., custom stages, source-database drivers, etc.)

Launch Redis Connect

Redis Connect includes scripts for launching a single instance. You can run the scripts as follows:

On Linux

redis-connect/bin$ ./redisconnect.sh    
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below
Check the value of redis.connection.url and credentials.dir.path in jobmanager.properties e.g.
redis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836
credentials.dir.path=/var/secrets/redis
*******************************
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------

On Windows

redis-connect\bin> redisconnect.bat

Starting an instance

Linux:

redis-connect/bin$ ./redisconnect.sh start

Windows:

redis-connect\bin> redisconnect.bat start

Swagger UI

Redis Connect Swagger UI is available on port 8282 by default. If you're running locally, you can point your browser to http://localhost:8282/swagger-ui/index.html

For quick start, use 'cdc_job' as jobName

Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use a sample job configuration: DB2, Files, Gemfire, MongoDB, MySQL, Oracle, PostgreSQL, SQL Server and VERTICA

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
For quick start, use 'stream' as jobType

Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus

Once you've configured a job, try inserting some records into the source database. Then confirm that they have arrived in Redis.

Copyright

Redis Connect is developed by Redis, Inc. Copyright (C) 2023 Redis, Inc.