fluent-plugin-redis-store icon indicating copy to clipboard operation
fluent-plugin-redis-store copied to clipboard

Add a congestion check as implemented in Logstash Redis output

Open maxgutman opened this issue 11 years ago • 2 comments

A congestion check prevents OOM errors when Redis is used as a broker/queue (list type). RedisStoreOutput will sleep until the list length is smaller than the specified threshold value. Ideally this check would happen prior to each message being pushed, but this can't happen during pipelining. Instead, the check happens for each chunk and there is a possibility of going above the threshold by the -1. Even so, this is still a very useful implementation that was taken from Logstash.

maxgutman avatar Nov 20 '14 18:11 maxgutman

Thank you for sending patch, and sorry for this delayed reply.

I have a question.

Why congestion_check() is called in write() instead of operation_for_list()? Is there any specific reason?

I think that while the check is only for the list operation, to call in the latter seems to be appropriate, and also in there it can the feature of key_path, not only key.

pokehanai avatar Mar 28 '15 04:03 pokehanai

@maxgutman any input?

sstarcher avatar Sep 20 '16 14:09 sstarcher