[Bug]: Job Schedulers Taking more space than Repeatable Jobs

[Haris-SP4RKy]

Version

v5.63.0

Platform

NodeJS

What happened?

🐞 Bug Report: Job Schedulers Taking More Space Than Repeatable Jobs

Summary: After migrating from Repeatable Jobs to the new Job Schedulers, we observed a significant increase in storage usage. Previously, with approximately 3 million repeatable jobs, the system maintained stable database storage usage. However, after upgrading to Job Schedulers, the storage footprint has roughly doubled, even though the number of scheduled jobs remains the same.

Environment:

Service: BullMQ (Job Scheduling System) Previous Version: Using Repeatable Jobs Current Version: Using Job Schedulers Job Count: 3,000,000 Database: Redis

Issue Description:

Under the previous implementation using Repeatable Jobs, Redis memory usage remained within acceptable limits for 3M jobs. After migrating to Job Schedulers, the same number of scheduled jobs now consumes approximately 2x more space. This increase appears to stem from the additional metadata or data structure changes introduced in the Job Scheduler model.

Impact:

Redis storage costs and memory pressure have significantly increased. Potential risk of reaching Redis memory limits sooner under the same load conditions. Could affect job throughput or cause slower performance under high scheduling volumes.

Expected Behavior: Memory usage for Job Schedulers should be comparable to or optimized relative to Repeatable Jobs for the same workload.

Observed Behavior: Job Schedulers occupy roughly double the space previously used by Repeatable Jobs under identical job counts and configurations.

How to reproduce.

Steps to Reproduce:

1. Create 3 million repeatable jobs using the older Repeatable Jobs feature and measure Redis memory usage.
2. Migrate to Job Schedulers and recreate the same set of jobs.
3. Compare Redis memory usage before and after migration.

Relevant log output

Code of Conduct

• [ ] I agree to follow this project's Code of Conduct

[manast]

Unfortunately this is working as design. In the previous versions of BullMQ, we did not store a meta object for every job scheduler, instead the scheduler was encoded in the repeatable job key itself. This was a dead end in terms of robustness and features so we had to move from that solution the one we have now. If you happen to have only 1 job per job scheduler at any given time (like you clean the jobs after completion for example), then yes, the effective increase in memory is going to be 2x. Whereas in a case where you keep maybe 100 or 1000 jobs for debug ability, the increase would just be 1% or 0.1%.