readyset

readyset copied to clipboard

ReadySet is a SQL caching engine designed to help developers enhance the performance and scalability of their existing database-backed applications.

• What is ReadySet?
• Getting Started
• Documentation
• Join the Community
• Development
• License

What is ReadySet?

ReadySet is a lightweight SQL caching engine that precomputes frequently-accessed query results and automatically keeps these results up-to-date over time as the underlying data in your database changes. ReadySet is wire-compatible with MySQL and Postgres and can be adopted without code changes.

ReadySet acts as both a SQL cache and a proxy– when you first connect ReadySet to your application, it defaults to proxying all of your queries to your backing database, so it doesn't change the behavior of your application.

From there, you can monitor the performance of your queries using the ReadySet dashboard, and cache a subset of them in ReadySet. ReadySet automatically keeps cached results up-to-date over time as the underlying data in your database changes, so there's no need to write cache maintenance logic.

Getting Started

Deploy ReadySet

The ReadySet orchestrator is a command line tool that uses Docker Compose to spin up a ReadySet instance on your local machine. The following command downloads and runs the ReadySet orchestrator:

bash -c "$(curl -sSL https://launch.readyset.io)"

The orchestrator will prompt you to select a wire protocol, and to connect ReadySet to an existing database (or create a new one).

Connect to ReadySet

• To connect to ReadySet via the MySQL or Postgres command line, run the command that the orchestrator prints after successfully deploying a ReadySet instance.
• To connect ReadySet to your application, use the ReadySet connection string (printed by the orchestrator after successfully deploying ReadySet) in the same way you would if it were a MySQL or Postgres connection string. Check out our docs for more information.

Once you connect to ReadySet, you can start issuing queries as you would with a traditional database.

Monitor Query Performance

By default, all queries sent to ReadySet are proxied to the backing database. You can see the current list of proxied queries and their performance profiles via the ReadySet dashboard which is accessible on http://localhost:4000.

Cache Queries

ReadySet supports caching both prepared statements (i.e. parameterized queries) and one-off queries. To cache a query in ReadySet, you’ll need either the query ID (which can be found in the dashboard, or by running SHOW CACHES) or the full query text (i.e. the SELECT statement).

From there, you can run CREATE CACHE FROM <query ID> or CREATE CACHE FROM <select statement> via the MySQL or Postgres client.

Write Handling

You can either send writes to ReadySet or directly to your backing database. If you send ReadySet a write, it will be proxied to your backing database. ReadySet waits to receive updates from your database's bin logs before updating the cached state to reflect those writes.

Note: The ReadySet Orchestrator is updated frequently with releases built off our internal ReadySet repository. Because of this, the orchestrator may have features not yet in our public repo. We're working on making our live codebase available on Github.

FAQs

Q: How does ReadySet work under the hood?

A: The heart of ReadySet is a query engine based on partially-stateful, streaming data flow. To learn more about how it works, see our docs.

Q: How does ReadySet keep cached state up to date?

A: ReadySet receives updates about data changes from your backing database via binlog replication and uses those updates to automatically update its internal state.

Q: Do I have to send all of my database traffic to ReadySet?

A: You can if you want to, but it’s not required. You can manually route a subset of your traffic to ReadySet (as you would with a traditional database read replica), or you can send all of it to ReadySet. It's important to note that not all of the queries sent to ReadySet need to be cached– you have fine-grained control over what is cached vs. what is proxied.

Documentation

For more information, check out our full docs site here.

Join the Community

For questions or support, join us on the ReadySet Community Discord. You can also post questions on our Github forum. Everyone is welcome!

Development

Prerequisites

Install Dependencies

Prior to running ReadySet, you may need to install the following dependencies: clang libclang-dev libssl-dev liblz4-dev build-essential

macOS:

brew install lz4
brew install [email protected]

Ubuntu:

sudo apt update && sudo apt install -y build-essential libssl-dev pkg-config llvm clang liblz4-dev cmake

Install Rust

ReadySet is written entirely in Rust. If you don’t already have Rust installed, you can install it via rustup (select the version of ‘nightly’ specified in the rust-toolchain file):

curl https://sh.rustup.rs -sSf | sh

Runtime Dependencies: Upstream Database and Consul

Local Development

To streamline local development, all runtime dependencies (Consul, MySQL, Postgres) can be run with:

cp docker-compose.override.yml.example docker-compose.override.yml
docker-compose up -d

Full Deployment

ReadySet runs alongside a backing MySQL or Postgres database and uses Consul for leader election. You’ll need to pass in your database’s connection string when running the ReadySet server and adapter, and have a Consul instance running.

Note: Consul persists data per deployment. Using the same deployment name will carry over any previously installed queries into a new deployment.

Run ReadySet

ReadySet Server

First, compile and run ReadySet server.

cargo run --bin readyset-server --features display_literals --release -- --replication-url <upstream-url>  --deployment <deployment name>

If using the databases supplied by the docker-compose environment in this repository, replace with the URL of the database corresponding to your database engine:

• MySQL: mysql://root:[email protected]/readyset
• PostgreSQL: postgresql://root:[email protected]/readyset

If running with an existing external database, replace with the connection string for that database.

ReadySet MySQL or Postgres Adapter

Then, run the adapter binary corresponding to your upstream database (MySQL or Postgres) The adapter will communicate with servers that have the same deployment name.

MySQL

cargo run --bin readyset-mysql --features display_literals --release -- --upstream-db-url mysql://root:[email protected]/readyset  --allow-unauthenticated-connections
  --address 0.0.0.0:3307 --deployment <deployment name>  --prometheus-metrics --query-log --query-log-ad-hoc

Postgres

cargo run --bin readyset-psql --features display_literals --release -- --upstream-db-url postgresql://postgres:[email protected]/readyset  --allow-unauthenticated-connections
  --address 0.0.0.0:5433 --deployment <deployment name> --prometheus-metrics --query-log --query-log-ad-hoc

The adapter listens for connections at the address specified in the address flag.

The query-log and query-log-ad-hoc flags ensure that queries are sent to the Prometheus client running in the adapter.

The prometheus-metrics flag exposes an HTTP endpoint in the adapter to allow querying of metrics. This can be reached with an HTTP GET request to :6034/prometheus (e.g., curl -X GET 127.0.0.1:6034/prometheus).

Testing

To run tests for the project, run the following command:

cargo test --skip integration_serial

Testing Notes

Certain tests cannot be run in parallel with others, and these tests are typically in files postfixed with _serial. Running the entire set of tests for a package, i.e. cargo test -p readyset-server may fail if serial tests are included.

Running tests may require increasing file descriptor limits. You can do so by running ulimit -Sn 65535.

Performance

When running ReadySet in a performance-critical setting, make sure you compile with the --release flag.

Note: This repository contains a snapshot of the current state of the ReadySet internal repository. Our team is currently working on getting our live codebase, including all development history, ready to be hosted on GitHub, at which point we'll force push to this repository.

License

ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. It is free to use on any number of nodes. The ReadySet team is hard at work getting the codebase ready to be hosted on Github.

ReadySet is also available as a paid cloud service that can be deployed to your accounts or hosted entirely by ReadySet. You can get early access to our cloud product by signing up here.