Why the drop count doesn't work ?

[cang233]

I write a demo to handle the packet to flow,the packet speed I receive is about 1.7M pps,but I ran this one night,there is the log ,the capture speed is low down but the drop info not print.

DEBUG: ---------------
DEBUG: System is using 4 cores now. 0 cores are left available.
DEBUG: Current speed of 0 instance of segment1 is 735544 PKT/S, cloneNumber: 1 queue number: 8
DEBUG: Current speed of 1 instance of segment1 is 685664 PKT/S, cloneNumber: 1 queue number: 8
DROP: slow operations mempool has less than 10% free space. This can lead to dropping packets while receive.
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: ------------------
DEBUG,ipv4-tcp-> mapper.map size: [1580271 ,1406791 ,1474459 ,1666465 ,1573361 ,]
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: ---------------
DEBUG: System is using 4 cores now. 0 cores are left available.
DEBUG: Current speed of 0 instance of segment1 is 450048 PKT/S, cloneNumber: 1 queue number: 8
DEBUG: Current speed of 1 instance of segment1 is 450560 PKT/S, cloneNumber: 1 queue number: 8
DROP: slow operations mempool has less than 10% free space. This can lead to dropping packets while receive.
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: ------------------
DEBUG,ipv4-tcp-> mapper.map size: [1581928 ,1408239 ,1475927 ,1668368 ,1574939 ,]
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
 (16)
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
 (16)
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
 (16)
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
 (16)
ERROR: AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
New packet Error:, AllocateMbuf cannot allocate mbuf, dpdk returned:  -105
DEBUG: ---------------
DEBUG: System is using 4 cores now. 0 cores are left available.
DEBUG: Current speed of 0 instance of segment1 is 9216 PKT/S, cloneNumber: 1 queue number: 8
DEBUG: Current speed of 1 instance of segment1 is 9728 PKT/S, cloneNumber: 1 queue number: 8
DROP: slow operations mempool has less than 10% free space. This can lead to dropping packets while receive.
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
DEBUG: Start new instance for segment1
WARNING: Can't start new clone for segment1 instance 2
...

you can see that the speed has low down from 1.7M pps to 20k pps,but there is still no drop info,why? my init code as present,and all work I do is in globalHandler function,just save the pkt to map according to the 5 tuples.

	config := flow.Config{
		CPUList:              "0-3",
		HWRXPacketsTimestamp: ifSupport,
	}
	flow.CheckFatal(flow.SystemInit(&config))

	inputFlow, err := flow.SetReceiver(0)
	flow.CheckFatal(err)
	flow.CheckFatal(flow.SetVectorHandler(inputFlow, globalHandler, nil))
	flow.CheckFatal(flow.SetStopper(inputFlow))
	flow.CheckFatal(flow.SystemStart())

[gshimansky]

You can try to enable additional debugging output if you uncomment lines with DEBUG and DEBUG_PACKET_LOSS in low.h: https://github.com/intel-go/nff-go/blob/master/internal/low/low.h#L50 You need to rebuild your application after that.

[cang233]

@gshimansky Ok,thanks.I'll try. But in nff-go/test/performance/per_gen.go,I can see the gen speed and the drop count,why I can not use this drop count in my code?

[gshimansky]

From your log I see that there is likely a problem with allocation. New packets cannot be created and this may mean that Receive doesn't receive any packets from the NIC. Such packets are not diagnosed as dropped because they aren't even received by software.

[cang233]

@gshimansky Not only there,but all of this log doesnt have the drop count info.Even at the starting of this program,there is enough mempool space ,but still no drop count.

[gshimansky]

In this case I suppose the problem is with allocation. All memory pools are overflowing and new mbufs cannot be created for received packets. What does your graph do? Are you storing packets somewhere? They all have to be either sent to dropped to allow memory reuse. You can also increase the default number of mbufs in memory pool by specifying MbufNumber parameter to SystemInit.

[cang233]

My packets are saved in memory,maybe jumped out of the nff-go graph. I can not statistics the packet info in the graph because the operations I do is too slow,so I handle it by myself.I save the information I need to a struct in nff-go handler and then handle it in my own function.

[aregm]

My packets are saved in memory,maybe jumped out of the nff-go graph. I can not statistics the packet info in the graph because the operations I do is too slow,so I handle it by myself.I save the information I need to a struct in nff-go handler and then handle it in my own function.

I am not sure I understand what you have done. If you can paste a code here we can help you.

[cang233]

All I do is in the globalHandler Function,to save packets in a map by its 5tuple.

	config := flow.Config{
		CPUList:              "0-3",
		HWRXPacketsTimestamp: ifSupport,
	}
	flow.CheckFatal(flow.SystemInit(&config))

	inputFlow, err := flow.SetReceiver(0)
	flow.CheckFatal(err)
	flow.CheckFatal(flow.SetVectorHandler(inputFlow, globalHandler, nil))
	flow.CheckFatal(flow.SetStopper(inputFlow))
	flow.CheckFatal(flow.SystemStart())

func globalHandler(ps []*packet.Packet, mask *[32]bool, context flow.UserContext) {

	var i uint64 = 0
	for i = 0; i < 32; i++ {
		if (*mask)[i] {
			ps[i].ParseL3()
			//ipv4
			tcp4, udp4, _ := ps[i].ParseAllKnownL4ForIPv4()
			if tcp4 != nil {
				p := &flows.Ipv4TCPPktInfo{
					Ipv4Hdr:     *ps[i].GetIPv4NoCheck(),
					TCPHdr:      *tcp4,
					TotalLen:    ps[i].GetPacketLen(),
				}
				flows.Ipv4TCPPktChan <- p
				continue
			} else if udp4 != nil {
				continue
			}

func save(p *Ipv4TCPPktInfo, flowMap map[string]*Ipv4TCPFlowInfo) {
	key := genKey(p)

	if v, ok := flowMap[key]; !ok || v == nil {
		//create
		flowMap[key] = &Ipv4TCPFlowInfo{
			StartTime: timeNow,
			EndTime:   timeNow,
			Pkts: []*Ipv4TCPPktInfo{
				p,
			},
		}
	}else{
               //update
        }