[@keyv/redis] Migrate to node-redis from ioredis

[jftanner]

trafficstars

Is your feature request related to a problem? Please describe. This was mentioned as part of https://github.com/jaredwray/keyv/issues/868, but not implemented as part of it. So this isn't new, but doesn't currently have an issue to track it.

With the effective deprecation of ioredis, devs are encouraged to switch to the official node-redis module instead. But, currently, that's not compatible with Keyv without a shim.

Describe the solution you'd like One of two options:

1. A new @keyv/node-redis module
2. Migrate @keyv/redis to use node-redis instead.

In the latter case, it would either be a breaking change or would require a shim to still accept an ioredis instance passed to the constructor (new KeyvRedis(redis);).

[jaredwray]

@jftanner - we will be migrating to that soon and do plan to make it a breaking change moving forward.

https://github.com/jaredwray/keyv/issues/1018

[jaredwray]

We are also considering using https://github.com/valkey-io/iovalkey to replace ioredis.

[jaredwray]

@100tomer

[100tomer]

Cool thanks for telling me about it. I already switched to another npm package but I'll be happy to switch back when it's ready.

[jaredwray]

Something to consider when looking at this. https://github.com/jaredwray/keyv/tree/ioredis-port-to-iovalkey

[jaredwray]

Cool thanks for telling me about it. I already switched to another npm package but I'll be happy to switch back when it's ready.

What library did you move too?

[jaredwray]

https://github.com/jaredwray/keyv/issues/1142

[jaredwray]

@jftanner @100tomer - Wanted to update you on the plan that we are doing for Redis moving forward. Since ioredis is no longer being supported we are going to split things into two storage adapters:

• @keyv/valkey - this is the open source redis server that is being supported by the linux foundation. We are in progress and will release it in the next couple weeks as it is based on the ioredis library now known as iovalkey. You can read more about Valkey by clicking on the link.
• @keyv/redis - we will be porting this over to node-redis and fixing some things such as namespace issues in October and November but will do a major version release with this change.

[jaredwray]

https://www.npmjs.com/package/@keyv/valkey has been released 🎉

@mcollina thanks for the work on porting this over!

[jaredwray]

@klippx - adding this in here: https://github.com/jaredwray/keyv/issues/1174

[jaredwray]

Just FYI as we are working on the new version and should have it out in the next 2-3 weeks.

[jaredwray]

@jftanner @100tomer @chocolateboy @klippx @leo-fresha - I am pleased to announce the updated KeyvRedis adapter that is based on node-redis aka https://www.npmjs.com/package/redis

Features

• Built on top of redis.
• TTL is handled directly by Redis.
• Supports Redis Clusters.
• Url connection string support or pass in your Redis Options
• Easily add in your own Redis client.
• Namespace support for key management.
• Unlink as default delete method for performance.
• Access to the Redis client for advanced use cases.
• Keyv and Redis Libraries are exported for advanced use cases.
• createKeyv function for easy creation of Keyv instances.
• jsDoc comments for easy documentation.
• CJS / ESM and TypeScript supported out of the box.

Table of Contents

• Usage
• Namespaces
• Performance Considerations
• High Memory Usage on Redis Server
• Using Cacheable with Redis
• Clustering and TLS Support
• API
• Migrating from v3 to v4
• About Redis Sets and its Support in v4
• License

Usage

Here is a standard use case where we implement Keyv and @keyv/redis:

import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379'));
keyv.on('error', handleConnectionError);

Here you can pass in the Redis options directly:

import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const redisOptions = {
  url: 'redis://localhost:6379', // The Redis server URL (use 'rediss' for TLS)
  password: 'your_password', // Optional password if Redis has authentication enabled

  socket: {
    host: 'localhost', // Hostname of the Redis server
    port: 6379,        // Port number
    reconnectStrategy: (retries) => Math.min(retries * 50, 2000), // Custom reconnect logic

    tls: false, // Enable TLS if you need to connect over SSL
    keepAlive: 30000, // Keep-alive timeout (in milliseconds)
  }
};

const keyv = new Keyv(new KeyvRedis(redisOptions));

Or you can create a new Redis instance and pass it in with KeyvOptions:

import Keyv from 'keyv';
import KeyvRedis, { createClient } from '@keyv/redis';

const redis = createClient('redis://user:pass@localhost:6379', { namespace: 'my-namespace'});
const keyvRedis = new KeyvRedis(redis);
const keyv = new Keyv({ store: keyvRedis });

Here is the same example but with the Keyv instance created with the createKeyv function:

import { createKeyv } from '@keyv/redis';

const keyv = createKeyv('redis://user:pass@localhost:6379', { namespace: 'my-namespace' });

You only have to import the @keyv/redis library if you are using the createKeyv function. 🎉 Otherwise, you can import Keyv and @keyv/redis independently.

Namspaces

You can set a namespace for your keys. This is useful if you want to manage your keys in a more organized way. Here is an example of how to set a namespace:

import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { namespace: 'my-namespace' }));

This will prefix all keys with my-namespace:. You can also set the namespace after the fact:

keyv.namespace = 'my-namespace';

NOTE: If you plan to do many clears or deletes, it is recommended to read the Performance Considerations section.

Performance Considerations

With namespaces being prefix based it is critical to understand some of the performance considerations we have made:

• clear() - We use the SCAN command to iterate over keys. This is a non-blocking command that is more efficient than KEYS. In addition we are using UNLINK by default instead of DEL. Even with that if you are iterating over a large dataset it can still be slow. It is highly recommended to use the namespace option to limit the keys that are being cleared and if possible to not use the clear() method in high performance environments.

• delete() - By default we are now using UNLINK instead of DEL for deleting keys. This is a non-blocking command that is more efficient than DEL. If you are deleting a large number of keys it is recommended to use the deleteMany() method instead of delete().

• clearBatchSize - The clearBatchSize option is set to 1000 by default. This is because Redis has a limit of 1000 keys that can be deleted in a single batch.

• useUnlink - This option is set to true by default. This is because UNLINK is a non-blocking command that is more efficient than DEL. If you are not using UNLINK and are doing a lot of deletes it is recommended to set this option to true.

• setMany, getMany, deleteMany - These methods are more efficient than their singular counterparts. If you are doing multiple operations it is recommended to use these methods.

If you want to see even better performance please see the Using Cacheable with Redis section as it has non-blocking and in-memory primary caching that goes along well with this library and Keyv.

High Memory Usage on Redis Server

This is because we are using UNLINK by default instead of DEL. This is a non-blocking command that is more efficient than DEL but will slowly remove the memory allocation.

If you are deleting or clearing a large number of keys you can disable this by setting the useUnlink option to false. This will use DEL instead of UNLINK and should reduce the memory usage.

const keyv = new Keyv(new KeyvRedis('redis://user:pass@localhost:6379', { useUnlink: false }));
// Or
keyv.useUnlink = false;

Using Cacheable with Redis

If you are wanting to see even better performance with Redis, you can use Cacheable which is a multi-layered cache library that has in-memory primary caching and non-blocking secondary caching. Here is an example of how to use it with Redis:

import KeyvRedis from '@keyv/redis';
import Cacheable from 'cacheable';

const secondary = new KeyvRedis('redis://user:pass@localhost:6379');

const cache = new Cacheable( { secondary } );

For even higher performance you can set the nonBlocking option to true:

const cache = new Cacheable( { secondary, nonBlocking: true } );

This will make it so that the secondary does not block the primary cache and will be very fast. 🚀

Clustering and TLS Support

If you are using a Redis Cluster or need to use TLS, you can pass in the redisOptions directly. Here is an example of how to do that:

import Keyv from 'keyv';
import KeyvRedis, { createCluster } from '@keyv/redis';

const cluster = createCluster({
    rootNodes: [
      {
        url: 'redis://127.0.0.1:7000',
      },
      {
        url: 'redis://127.0.0.1:7001',
      },
      {
        url: 'redis://127.0.0.1:7002',
      },
    ],
});

const keyv = new Keyv({ store: new KeyvRedis(cluster) });

Here is an example of how to use TLS:

import Keyv from 'keyv';
import KeyvRedis from '@keyv/redis';

const tlsOptions = {
    socket: {
      host: 'localhost',
      port: 6379,
      tls: true,  // Enable TLS connection
      rejectUnauthorized: false,  // Ignore self-signed certificate errors (for testing)
      
      // Alternatively, provide CA, key, and cert for mutual authentication
      ca: fs.readFileSync('/path/to/ca-cert.pem'),
      cert: fs.readFileSync('/path/to/client-cert.pem'),  // Optional for client auth
      key: fs.readFileSync('/path/to/client-key.pem'),    // Optional for client auth
    }
};

const keyv = new Keyv({ store: new KeyvRedis(tlsOptions) });

API

• constructor([connection], [options])
• namespace - The namespace to use for the keys.
• client - The Redis client instance.
• keyPrefixSeparator - The separator to use between the namespace and key.
• clearBatchSize - The number of keys to delete in a single batch.
• useUnlink - Use the UNLINK command for deleting keys isntead of DEL.
• set - Set a key.
• setMany - Set multiple keys.
• get - Get a key.
• getMany - Get multiple keys.
• has - Check if a key exists.
• hasMany - Check if multiple keys exist.
• delete - Delete a key.
• deleteMany - Delete multiple keys.
• clear - Clear all keys. If the namespace is set it will only clear keys with that namespace.
• disconnect - Disconnect from the Redis server.
• iterator - Create a new iterator for the keys.

Migrating from v3 to v4

The main change in v4 is the removal of the ioredis library in favor of the @keyv/redis library. This was done to provide a more consistent experience across all Keyv storage adapters. The @keyv/redis library is a wrapper around the redis library and provides a more consistent experience across all Keyv storage adapters. The only other change is that we no longer do redis sets as they caused performance issues.

About Redis Sets and its Support in v4

We no longer support redis sets. This is due to the fact that it caused significant performance issues and was not a good fit for the library.

[klippx]

Allowing redis sets might be one of the the biggest mistake in this library's history, for us it has caused nothing but grief. Good riddance!

[jaredwray]

@klippx - agreed and most likely we will remove it when we move @keyv/valkey to the next major release.