Dynamic sampling rate for 10Gbps

[nidotech]

I've a problem with very inaccurate sampling rate for 10Gbps link with low traffic (less than 500Mbps). Is it possible to implement a dynamic adjusted sampling rate into hsflowd?

[sflow]

That approach leads to statistical bias in the results. However the analysis is usually not very sensitive to sampling rate, so can you explain more about the problem? If your application really needs the finer granularity, then why not set a more aggressive sampling rate? The default for 10G would be 1-in-10000, but in /etc/hsflowd.conf you can set something like:

sampling.10G = 5000

[nidotech]

Since i'm measuring the traffic on a edge device, i've changed to the following rules in order to take care of sampling instead of pcap. However, it is still very inaccurate.

*raw -A PREROUTING -j NOTRACK -A OUTPUT -j NOTRACK

-A FORWARD -i enp101s0f0 -m statistic --mode random --probability 0.0025 -j NFLOG --nflog-group 1 --nflog-prefix SFLOW -A FORWARD -o enp101s0f0 -m statistic --mode random --probability 0.0025 -j NFLOG --nflog-group 1 --nflog-prefix SFLOW

/etc/hsflowd.conf polling = 20 nflog { group = 1 probability = 0.0025}

[sflow]

To run a calibration test, it works well to set up a constant flow with a moderate traffic level. For example, if there is a target IP you can ping, then something like this will generate 100 packets/sec in each direction:

sudo ping -i 0.01 <targetIP>

With this you should see a sample from that flow every 4 seconds (on average) or around 15 samples/min. If you don't see that then there must be (1) a bottleneck where samples are dropped or (2) a problem with nflog or (3) a bug in hsflowd.

For a more comprehensive calibration-check I recommend the sflow-test app in sFlow-RT: https://blog.sflow.com/2015/11/sflow-test.html

Let me know how it goes.

[nidotech]

Does the tcp offload or other nic offload functions affecting the sampling collection?

[sflow]

Yes, I would expect TCP-offload to affect it. Are you seeing 64KB packet-samples, or do the samples disappear altogether?

If you suspect that hsflowd is rejecting the samples somehow, then running with debug logging might shed light:

sudo systemctl stop hsflowd sudo hsflowd -dddd 2>&1

The above will print lines of output for every sample that hsflowd receives.