revalidateTag is very slow when there are lots of keys

[zipme]

Brief Description of the Bug When using redis-string, and once redis has lots of keys (we have 18745) the revalidateTag could take 10 seconds to run

Severity [Major]

Frequency of Occurrence [Always]

Steps to Reproduce Provide detailed steps to reproduce the behavior, including any specific conditions or configurations where the bug occurs:

1. Populate redis with more than 10000 keys
2. Call revlidateTag

Expected vs. Actual Behavior The revalidation should hopefully take less than a second but it takes way longer (sometime 10s+)

Environment:

• OS: [Linux]
• Node.js version: [e.g., v20.11.0]
• @neshca/cache-handler version: [e.g., 1.3.2]
• next version: [e.g., 14.2.4]

[better-salmon]

@zipme, hello! Which Next.js router do you use: App or Pages?

[zipme]

Hi @better-salmon we use app router. When the cache is not primed the revalidation goes fast.

[better-salmon]

Thanks for bringing this issue to our attention. I understand the frustration caused by the long execution time of the revalidateTag process, especially with a significant number of keys in Redis.

The current revalidation process has algorithmic flaws. It uses an O(n²) algorithm to find all cache entries needing revalidation. This issue is inherited from Next.js, which recommends using this approach to delete cache entries. The implementation must use Array.prototype.includes inside a loop, which results in the observed performance issues. Here's the code from the Next.js docs:

revalidateTag(tag) {
    // Iterate over all entries in the cache
    for (let [key, value] of cache) {
        // If the value's tags include the specified tag, delete this entry
        if (value.tags.includes(tag)) {
            cache.delete(key)
        }
    }
}

Unfortunately, I don't see a way to improve this due to how Next.js uses tags.

I've made some optimizations in the latest pull request to address Redis's performance concerns. A new option, revalidateTagQuerySize, has been added to the redis-stack and redis-strings Handlers. This option allows you to specify the number of tag lists retrieved in a single query from Redis during the scanning or searching. Additionally, the default query size for hScan in redis-strings and ft.search in redis-stack has been increased from 25 to 100. This adjustment reduces the number of network roundtrips by optimizing the number of commands sent to Redis, though it increases the TCP packet size. By increasing the query size, we aim to balance command count and network roundtrips, improving performance in scenarios with many keys. In addition, I've made the updating of the sharedTags map command isolated, and it must not block other Redis commands.

Finding the best revalidateTagQuerySize for your setup can vary based on your environment and workload. Start with a value of 100 and adjust it while monitoring the performance impact. Experiment with different values to fine-tune the balance between the number of commands sent to Redis and the size of the TCP packets.

Please update to the version 1.4.0 or newer with these optimizations and test if the performance improves in your setup. Your feedback will be valuable in further refining the solution.

Thanks for your patience and understanding.

[zipme]

@better-salmon how about creating a tags manifest? Something like this maybe?

[better-salmon]

I used to utilize the tags manifest in versions before 1.0.0, but I need time to recall why I stopped using it.

[TimGeudens]

We also saw a big performance impact when revalidating tags using the redis-strings handler. In our case the hash holding the mapping of all cache tags was holding about 100k items, so each revalidation did 1000 queries to Redis.

We opted to write a custom cache handler (based on redis-strings) which uses sets to fetch the tag mapping: const setTagsOperation = cacheHandlerValue.tags.length ? cacheHandlerValue.tags.map(tag => { return client.sAdd(sharedTagsKey + tag, key); }) : undefined;

Pretty easy (and performant) to fetch the tags then: const keys = await client.sMembers(sharedTagsKey + tag);

This change made a huge impact on our system.

[Antonio-Laguna]

@TimGeudens very long shot but any chance you could share that? Being hit by this now!

[TimGeudens]

@Antonio-Laguna we kept it quite easy: In the set handler:

async set(key, cacheHandlerValue) {
...#custom code, like gzipping
const setOperation = client.set(
          options,
          keyPrefix + key,
          JSON.stringify(cacheHandlerValue),
          expireAt,
        );
        const setTagsOperation = cacheHandlerValue.tags.length
          ? cacheHandlerValue.tags.map(tag => {
              return client.sAdd(sharedTagsKey + tag, key);
            })
          : undefined;
        await Promise.all([setOperation, setTagsOperation]);
}

and then in the revalidateTag handler:

async revalidateTag(tag) {
 const keys = await client.sMembers(sharedTagsKey + tag);
      if (!keys.length) {
        return;
      }
      const keysWithPrefix = keys.map(key => keyPrefix + key);
      const deleteKeysOperation = client.unlink(
        commandOptions({ signal: AbortSignal.timeout(timeoutMs) }),
        keysWithPrefix,
      );
      const deleteTagsOperation = client.del(sharedTagsKey + tag);
      await Promise.all([deleteKeysOperation, deleteTagsOperation]);
}

This creates a set for each tag which makes it much more performant to do the lookup. You'll need to setup the get too but that's quite straightforward.

[Antonio-Laguna]

Appreciate it Tim!

[summerua]

@TimGeudens Thanks for sharing this! Have you considered publishing your own package or creating a separate handler to address that issue?

[tilman]

As it seems this plugin is not actively maintained anymore we created a new improved redis cache handler for next.js to fix this issue. It eliminates the resource hungry hscan by including it via some in-memory caching synced via keyspace notification (so it is multi node capable).

You can take a look here: https://github.com/trieb-work/nextjs-turbo-redis-cache