dragonfly

dragonfly copied to clipboard

Pipeline with many threads becomes really slow. 5M entries take:

• 1+s on 1 thread
• 4s on 4 threads
• 7s on 8 threads 💀

For comparison:

• Redis takes ~1 s
• MULTI/EXEC with squashing takes about ~1s (a little less)

While the new sqashing is the fastest of them all, it requries all the commands to be in memory ahead. Non-atomic squashing doesn't require it - but makes locking more difficult.

My only suggestion would be squashing without a carrier & stub transaction - i.e. just sharding and spreading the load over threads and doing a full dispatch each time.

If #1036 is fixed, inline scheduling would also greatly improve performance here.

Can we close this @dranikpg ?