axosyslog icon indicating copy to clipboard operation
axosyslog copied to clipboard

Scalable disk queue

Open sodomelle opened this issue 1 month ago • 4 comments

@bazsi did a bunch of changes. The most important one in our case is that he changed from g_lists to ivykis list for in-memory queues in LogQueueNonReliableDisk.

@bshifter and I added a new output queue, that takes batches of messages instead of a single message, to avoid frequent locking and unlocking.

sodomelle avatar Nov 25 '25 15:11 sodomelle

I'd like to review this too.

MrAnno avatar Nov 26 '25 09:11 MrAnno

As I've heard the code builds on front_cache_size > 0, but nothing ensures this, the user can set it to 0. Shouldn't we have an override for == 0?

alltilla avatar Nov 26 '25 14:11 alltilla

Done with another round of review, I won't go another one until we process my comments so far.

alltilla avatar Nov 26 '25 15:11 alltilla

Current implementation involves the possibility of requiring more memory due to front_cache and front_cache_output both having the full front_cache_size length. If we decide it is too much, or not intended, we can assign half the size to front_cache, and the other half to front_cache_output.

sodomelle avatar Nov 27 '25 15:11 sodomelle

Thank you for the fixes!

alltilla avatar Nov 28 '25 17:11 alltilla