nprobe and elasticsearch

[papage]

Hi all,

I have nprobe exporting to ELK but I am having trouble to index more than 1300 docs per sec. It is not an ELK issue. Since elasticsearch is multithreaded and says that "In order to use all resources of the cluster, you should send data from multiple threads or processes.", I believe nprobe should open multiple threads to Elasticseach, maybe to different cluster members.

So, I have a few questions about that:

• Does nprobe open mutiple connections to Elasticsearch?
• How can I see how many flows (ELK documents) are lost per sec and how many are really indexed?
• Is there a suggested ELK setup in order to absorb a 10K docs/s load?
• I can see in ELK ( documents per 30sec) and ntopng that performance is not steady (nothing to do with variations in traffic). Many times it drops to zero for a minute or two which is weird since CPU is not saturated,
• What should it be, the behaviour of nprobe, when it can't cope with the load? How do I know if it can't cope with the load? CPU goes to max?

Keep in mind that I am doing the tests in an non-licenced nprobe with pfring and not pfring_zc, but I had the same feeling when i did the tests in a licencsed ntopng with pfring_zc, a year ago.

Thanx, Sp