EPS degrade when the number of workers is four times of CPU cores

[kaisecheng]

In the benchmark comparing persistent queue (PQ) and memory queue, the EPS for 8 and 16 workers is similar. However, with 32 workers, the EPS for PQ significantly degrades compared to the memory queue. PQ does not receive enough events despite having available CPU resources. This suggests that thread contention, where worker threads block the write thread, may be the issue.

The logstash instance has 8 CPU cores and 8G memory.

CPU	MEM	Worker	Batch Size	Queue	1min EPS	5min EPS	Worker Utilization	Worker Concurrency	Avg CPU%	Avg Heap	Avg Non Heap	Q Max Events	Q Max Size
8	8	8	1000	persisted	6172	6094	96.32	7.705	28	1733266033	181894718	168347	1073787947
8	8	16	1000	persisted	9790	9514	83.56	13.37	47	2356683605	183512033	797	132697837
8	8	32	1000	persisted	9253	9170	44.11	14.11	46	2463494258	184405256	30	147625218
8	8	8	1000	memory	5823	5585	99.99	8	21	2242772082	179471492	0	0
8	8	16	1000	memory	10230	9766	99.98	16	45	1976006473	178854446	0	0
8	8	32	1000	memory	16000	15550	99.99	32	71	2394150426	180885376	0	0