rump

rump copied to clipboard

Hot sync two Redis databases using dumps.

Why

There's no easy way to sync data from an AWS ElastiCache or GCP MemoryStore Redis cluster; the standard commands BGSAVE and SLAVEOF are blocked.

Rump is able to live sync Redis databases across cloud providers by only using SCAN, DUMP and RESTORE.

It's used at Sticker Mule to keep staging and development environments in sync with the production AWS/GCP Redis clusters.

Examples

# Sync local Redis DB 1 to DB 2.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2

# Sync ElastiCache cluster to local.
$ rump -from redis://production.cache.amazonaws.com:6379/1 -to redis://127.0.0.1:6379/1

# Sync protected ElastiCache via EC2 port forwarding.
$ ssh -L 6969:production.cache.amazonaws.com:6379 -N [email protected] &
$ rump -from redis://127.0.0.1:6969/1 -to redis://127.0.0.1:6379/1

# Dump GCP MemoryStore to file.
$ rump -from redis://10.0.20.2:6379/1 -to /backup/memorystore.rump

# Restore backup to ElastiCache.
$ rump -from /backup/memorystore.rump -to redis://production.cache.amazonaws.com:6379/1

# Sync with verbose mode disabled.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2 -silent

# Sync with TTLs.
$ rump -from redis://127.0.0.1:6379/1 -to redis://127.0.0.1:6379/2 -ttl

Features

• Uses SCAN instead of KEYS to avoid DoS servers.
• Doesn't use any temp file.
• Can sync any key type.
• Can optionally sync TTLs.
• Uses buffered channels to optimize slow source servers.
• Uses implicit pipelining to minimize network roundtrips.
• Supports two-step sync: dump source to file, restore file to database.
• Supports Redis URIs with auth.
• Offers the same guarantees of the SCAN command.

Demo

Development

# requirements: docker, docker-compose (dc)
dc up # watch/run Rump tests and a Redis container
dc run --rm rump sh # get shell on Rump container
dc run --rm redis sh; redis-cli -h redis # get Redis console

Install

Binaries can be found on the releases page.

curl -SL https://github.com/stickermule/rump/releases/download/1.0.0/rump-1.0.0-linux-amd64 -o rump \
  && chmod +x rump;
./rump

You can run rump in a container following the Dockerfile.example.

Mentions

• Sticker Mule Blog
• The Changelog
• Go Weekly
• DB Weekly
• Cron Weekly
• Hacker News
• Redis Weekly

Maintainers

nixtrace

Collaboration & License

• Contributing
• Code of Conduct
• MIT License