Cortex Rate Limit seems not to work

[Passimist]

I set a rate limit of 4 requests per minute for the Virus Total Get Report analyzer. (Because this is the free API limit) When I now mark 8 URL observables in TheHive and start the analyzer I expected cortex to analyze 4 of them and queue the others in order to analyze those after one minute. Instead all 8 jobs are started instantly and 4 of them fail because of the API limit.

Am I using this feature wrong or is it not working? This is my setup:

Any help is appreciated!

[mnmnc]

I don't have a solution but a suggestion for further investigation to narrow down the cause: Change API limit to 2 and see if it executes 4. If so then I would point to application.conf section on analyzer.fork-join-executor as the cause, since its possible that limit is not considering the parallelism.

[Passimist]

I've set the API Limit to 2 and selected 4 observables in TheHive and then started the analyzer for those 4 via the TheHive webinterfaces drop down menu "Run analyzers". Then I checked in Cortex jobs history where all 4 jobs finish in about 5 seconds.

This are my application.conf settings:

I changed parallelism-max to 2. Now when I repeat the same test it only starts 2 jobs but sadly the remaining 2 are not queued and processed after one minute. They just disappear.

[mnmnc]

Are you sure they disappear? Remember that depending on cache value if you run a job for a value that is in cache, the new job will not be created. Instead cortex responds with an ID of job from cache.
If they really disappear - thats unusual.

[Passimist]

I have the cache set to 0 minutes (so no caching at all I think?).

When selecting multiple observables in TheHive and starting the analyzer for them all at once the rate limit seems to be ignored. When I start the analysis of the different observables quickly one by one I recieve an error message mentioning the rate limit. Same error occurrs for analyses started via thehive4py. When I select multiple observables and start the analyzer for all at once when the rate limit is already reached there is no such error message displayed (this is what looked to me like analyses disappearing).

I personally would prefer to queue any analysis after the rate limit is reached. Maybe a config option to decide if i want them dropped or queued would be cool.