CB to pass to no-cache mode when csub-cache has a dangerous memory size

[fgalan]

It has been tested that when the number of subscriptions in the csubs cache is very large (300000 csubs), they it can exhaust the system memory an the CB crashes. This issue doesn't happen when -noCache is active.

CB should implement a protection mechanism, so if the csub cache size is very large (a CLI parameter will define the limit) it passes to nocache mode until the number of csub number (and cache memory size) decreases undel the limit.

Note that 300000 subscriptions is 3 orders of magnitude greater than current uses cases. Thus, the issue is not very probable in a real case (it is rather a corner case) but a protection measure could be developed anyway.

[fgalan]

Joining issue https://github.com/telefonicaid/fiware-orion/issues/1634 (now closed), which is about this same thing:

Current csubs cache stores all subscriptions, so it is a full mirror of DB subscriptions. However, when the number of csubs becomes large (e.g. 1,000,0000 subscriptions) it have a severe footprint on memory (it could be even crash CB).

This issues is about implementing a best approach, e.g. LRU policy (or other https://en.wikipedia.org/wiki/Cache_algorithms). Details have to be elaborated.

[fgalan]

Documentation at section https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/known_limitations.md#subscription-cache-limitation should be updated to remove the reference to this issue, once completed.

[kzangeli]

Instead of hardcoding a 'critical' number of issues, I'd propose to implement a very simple algorithm, based on the available RAM in the host, to calculate the number of issues for when to disable the sub-cache

[fgalan]

Both could make sense, depending on the use case. Ideally, something like:

-csubCacheLimit 100000

or

-csubCacheLimit adaptative

would be the best.

[kzangeli]

Sure. I'd set default value to 'adaptative' (a value of 0). If -csubCacheLimit is used on command line, it overrides the default calculated value

[Anjali-NEC]

Hi @fgalan @kzangeli Sir,

I would like to work on this issue. Please let me know this issue still need to be fix?

[Anjali-NEC]

Hi @fgalan sir,

As per my understanding of this issue we need to add a CLI -csubCacheLimit for defining the limit. when the csub cache size is greater then the -csubCacheLimit CB to pass to no-cache mode.

Please confirm my understanding.

[fgalan]

Hi @fgalan sir,

As per my understanding of this issue we need to add a CLI -csubCacheLimit for defining the limit. when the csub cache size is greater then the -csubCacheLimit CB to pass to no-cache mode.

Please confirm my understanding.

That would correspond with the simpler case (i.e. based in a fixed value). The other case (make it adaptative) is much complex.

However, although somehow valid, this issue is not a very interesting to implement. The cache size doesn't use to be a problem and, when it is, it's better to increase the CB memory than switch to no-cache mode (performance difference can be high, taking into account that the number of csubs would be very high in this case, if they don't fit in memory).

In sum: I'd suggest to pick another issue.