[ERROR: MBufRawPacket.cpp: init:55 ] Couldn't allocate mbuf

What typical errors of code specifics goes to this mbuf error ?

[ERROR: MBufRawPacket.cpp: init:55 ] Couldn't allocate mbuf

cat /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

Y

cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

2048

cat /proc/meminfo | grep Huge

AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB HugePages_Total: 2048 HugePages_Free: 1951 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 4194304 kB

```sh
sudo lib/pcpp/setup_dpdk.py status

Network devices using kernel driver
===================================
0000:01:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp1s0np1 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:06:12.0 'Virtio network device 1000' if=enp6s18 drv=virtio-pci unused=igb_uio,vfio-pci,uio_pci_generic *Active*

• first try to look on memory leaks:

sudo valgrind bin/send

==3066== Using Valgrind-3.22.0 and LibVEX; rerun with -h for copyright info
==3066== Command: bin/send
==3066== 
arg[0] = <bin/send>

net:init
        cores:32
        cmask:11111111111111111111111111111111

ERROR: This system does not support "SSE4_1".
Please check that RTE_MACHINE is set correctly.
EAL: unsupported cpu type.
[ERROR: DpdkDeviceList.cpp: initDpdk:143             ] failed to init the DPDK EAL
send: /home/dev01/pcapp_based/src/net.cpp:15: static void Net::init(bool): Assertion `pcpp::DpdkDeviceList::initDpdk(Net::coreMask, MBUF_POOL_SIZE)' failed.

You'll see this error when rte_pktmbuf_alloc returns nullptr which probably indicates that the mempool used to allocate packets is empty. What's the mempool size you're using in initDpdk()?

I make this tests:

• this was default and worked just one week ago before NIC reinstall
  • #define MBUF_POOL_SIZE (0x10000 - 1)
• this tests also shows the same behavour:
  • #define MBUF_POOL_SIZE (0x20000 - 1)
  • #define MBUF_POOL_SIZE (0x100000 - 1)

assert(pcpp::DpdkDeviceList::initDpdk(Net::coreMask, MBUF_POOL_SIZE));

• latest log with bug:

arg[0] = <bin/send>

net:init
	cores:32
	cmask:11111111111111111111111111111111

EAL: Detected CPU lcores: 32
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
VIRTIO_INIT: eth_virtio_pci_init(): Failed to init PCI device
PCI_BUS: Requested device 0000:06:12.0 cannot be used

open:0
	DPDK_0 mtu:1500
arg[1] = <etc/config.json>
arg[2] = <etc/rift_cfg_631.json>

config:run
	sensor:S_1_1 #2531001
		check:ok
	core:8
garp:

	slicer:S_1_1
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf
[ERROR: MBufRawPacket.cpp: init:55                   ] Couldn't allocate mbuf

• hugepages ok

$ cat /proc/meminfo | grep -i huge

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    2048
HugePages_Free:     2026
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         4194304 kB

• this config was forced to use after NIC replacement,

$ sudo lib/pcpp/setup_dpdk.py status

Network devices using kernel driver
===================================
0000:01:00.0 'MT27800 Family [ConnectX-5] 1017' if=enp1s0np1 drv=mlx5_core unused=igb_uio,vfio-pci,uio_pci_generic 
0000:06:12.0 'Virtio network device 1000' if=enp6s18 drv=virtio-pci unused=igb_uio,vfio-pci,uio_pci_generic *Active*

maybe I need some special DPDK-only code to initialize and try to send few raw packets just to get more detailed info dircetly from DPDK status functions & returns values on every calls?

@ponyatov it looks like you're pre-allocating enough mbufs, but still the system runs out of mbufs... are you sure the hugepages size is enough for these mbufs?

Also - there is probably a leak in your code (or a bug in PcapPlusPlus) that prevents these mbufs from being released. In order to debug I'd start with a very simple loop that captures packets and releases them right after - if this works you can slowly make the code more complex to understand where the problem is