Ability to limit size of job data

[gabegorelick]

Feature request

Many queues have upper limits for the size of messages that you can send through them. For example, SQS limits messages to 256 KB.

Such limits are really important for a queue backed by Redis, since memory may be limited: you don't want someone enqueuing 1GB jobs!

Bull should check the size of jobData and return an error if it's larger than a configurable limit: https://github.com/OptimalBits/bull/blob/c0ee6be03c6d47d60465d2bae0d2566d5bf0d661/lib/job.js#L61-L64

Since it has to serialize job data to a string anyway, Bull should be able to approximate the size a job would occupy in Redis by checking the length of the serialized string: https://github.com/OptimalBits/bull/blob/c0ee6be03c6d47d60465d2bae0d2566d5bf0d661/lib/job.js#L132-L135

[joshterrill]

I can give this a shot if that's okay. Seems to be a fairly simple introduction into getting my feet wet with contributing to the repo.

[joshterrill]

@gabegorelick Would you like the default size to be set to 256kb in the setDefaultOpts method?

[gabegorelick]

Would you like the default size to be set to 256kb in the setDefaultOpts method

256 KB is fine by me, but wouldn't setting that as a default be a breaking change?

[joshterrill]

Yeah I was just thinking about that. So I won't have it set a default, but I'll check for the existence of the sizeLimit, if it exists and the data goes over the limit, I can throw an exception.

[joshterrill]

@gabegorelick I can either let the user define the sizeLimit in bytes or I can implement a small library like https://www.npmjs.com/package/bytes and create a utility method which will let the user either enter a number in bytes, or a string such as 1mb or 1gb and that library will automatically convert it to bytes. I don't know what the rule is for adding extra dependencies, but this one is pretty small - if you want to look at the source code for it, it's here: https://github.com/visionmedia/bytes.js/blob/master/index.js

[gabegorelick]

I can either let the user define the sizeLimit in bytes or I can implement a small library like https://www.npmjs.com/package/bytes

I ultimately will defer to @manast and other core Bull contributors, but I would just accept a Number of bytes. If users want to use a third party library so that they can declare it as a string that's fine, but then they can bring their own preferred lib.

EDIT: looks like this was indeed the conclusion decided in https://github.com/OptimalBits/bull/pull/1378#discussion_r302279499