cloudamqp
Reference count segments and delete when zero references
Way faster than collecting references and hole punch segments. But of course not as space efficent. Can be somewhat mitigated with smaller segment sizes, and in the future compact segments (somehow).
Looks reasonable, and I've been running this with some synthetic traffic for a day without issues. I'm thinking about exceptions, if there's a risk of missing a inc/dec in case of an unexpected error, but haven't fully thought this through if it's possible.
Fixed the durable exchange problem. It's much faster consuming a 100M msgs queue now. Previously bytesize was recaluclated each time, now we have a counter that keeps adds/substracts when messages are added to the ready deque
The only problem is that it's "relatively" easy to exhaust disk space now, as now compaction of individual segments is being made. Publish and consume from one queue very fast, only publish to another queue very slowly. No segments will be deleted as at least one message from the slow queue is in each segment.
todo: when consuming with manual ack and then with auto ack sometimes Unexpected exception in deliver_loop: Missing key 7 (KeyError) can happen, or Invalid memory access (signal 11) at address 0x7fb15cd45fed
bin/avalanchemqperf throughput -z 5 -a
bin/avalanchemqperf throughput -z 5
Fixed now in https://github.com/cloudamqp/avalanchemq/pull/304/commits/53491c42ba5b814404b4a7dd94a70123af972cbd
Big question is if only deleting segments when no messages resides in them is enough or if too much disk space will be wasted...