netbenches

FreeBSD network forwarding performance benchmark results

This repository includes bench scripts (need netmap pkt-gen and nanobsd like), bench configuration files, raw results and gnuplot graphs on few setups. Bench script documentation is available too.

Hardware benched are:

SuperMicro X10SRW-F (16 cores Intel Xeon E5-2697A v4 @ 2.60GHz) and: 40G Chelsio T580-LP-CR, 100G Mellanox ConnectX-4 MCX416A-CCAT
Dell PowerEdgre R630 (2x12 cores Intel Xeon E5-2650 v4 @ 2.2GHz) and: Intel 82599, Chelsio T520-CR, Mellanox ConnectX-3 and ConnectX-4 (MCX4121A-XCA)
HP ProLiant DL360p Gen8 (8 cores Intel Xeon E5-2650 @ 2.60GHz) and 10-Gigabit Chelsio T540-CR
SuperMicro X9SRL-F (10 cores Intel Xeon E5-2650L v2 @ 1.70GHz) and 10-Gigabit Chelsio T540-CR
SuperMicro 5018A-FTN4 (8 cores Atom C2758 @ 2.40GHz) and 10-Gigabit Chelsio T540-CR
SuperMicro 5018A-FTN4 (8 cores Atom C2758 @ 2.40GHz) and 10-Gigabit Intel 82599
Netgate RCC-VE 4860 (4 cores Intel Atom C2558 @ 2.40GHz) and quad port Intel i350 (the dual port Intel i211 are not benched)
PC Engines APU2C4 (4 cores AMD GX-412TC @ 1GHz) and 4 Intel i210AT Gigabit NIC

Specific benches:

Impact of firewalls (ipfw/pf/ipf) on forwarding performance on:
- 16 cores Xeon E5-2697A & 100G Mellanox ConnectX-4 MCX416A-CCAT
- 16 cores Xeon E5-2697A & 40G Chelsio T580
- 2x12 cores Xeon E5-2650 & 10G Mellanox ConnectX-4 Lx
- 2x12 cores Xeon E5-2650 & 10G Chelsio T520-CR
- 2x12 cores Xeon E5-2650 & 10G Intel 82599ES
- 10 cores Xeon E5-2650L & 10G Chelsio T540
- 8 cores Xeon E5-2650 & 10G Chelsio T540
- 8 cores Atom C2758 & 10G Chelsio T540
- 8 cores Atom C2758 & 10G Intel 82599
- 4 cores Atom C2558 & Intel i350
- 4 cores AMD GX-412TC & Intel i210
- (old)4 cores Xeon L5630
- (old)2 cores AMD G-T40E
One year evolution:
- One year (2017) of forwarding performance with 8 cores Xeon E5-2650 & Chelsio T540
- One year (2017) of forwarding performance with 4 cores Atom 2558 & Intel i350
- One year (2016) of forwarding performance with 8 cores Atom 2758 & Chelsio T540
- One year (2016) of forwarding performance with 4 cores Atom 2558 & Intel i350
- One year (2016) of forwarding performance with 4 cores AMD GX-412TC & Intel i210
- One year (2015) of fastforwarding performance evolution with with 8 cores Xeon E5-2650
- One year (2015) of fastforwarding performance evolution with 4 cores Atom C2558
Hyper Threading impact:
- 16 cores Xeon E5-2697A & 100G Mellanox ConnectX-4
- 10 cores Xeon E5-2650L & Chelsio T540
- 8 cores Xeon E5-2650 & Chelsio T540
IPFW-at-NIC-level (DDoS protection)
- 16 cores Xeon E5-2697A & 100G Mellanox ConnectX-4
- 16 cores Xeon E5-2697A & 100G Chelsio T580
- 10 cores Xeon E5-2650L & Chelsio T540
comparing 12-head and 11-stable:
- fbsd12-head vs 11-stable r318516 with 4 cores AMD GX-412TC
comparing 11.0 and 11.1-BETA1:
- FreeBSD 11.0 vs 11.1-BETA1 on a 8 cores Xeon E5-2650 with Chelsio T540
- FreeBSD 11.0 vs 11.1-BETA1 on a 4 cores AMD GX-412TC with Intel i210
Impact of entropy sources on FreeBSD 11.1 forwarding performance:
- 8 cores Xeon E5-2650
- 8 cores Atom C2758
- 4 cores Atom C2558
- 4 cores AMD GX-412TC
Impact of adding VIMAGE support (without using it) on FreeBSD 11.1 forwarding&firewalling performance:
- 8 cores Xeon E5-2650
- 8 cores Atom C2758
- 4 cores Atom C2558
- 4 cores AMD GX-412TC
- 4 cores Xeon L5630
Impact of Intel's TurboBoost on FreeBSD 11-STABLE r327668:
- 8 cores Atom C2758 & Chelsio T540
- 8 cores Atom C2758 & Intel 82599
Impact of if_bridge on FreeBSD 11.1:
- 8 cores cores Xeon E5-2650
- 8 cores Atom C2758 & Chelsio T540
Impact of enabling ALTQ on FreeBSD 11-STABLE r332393:
- 4 cores Atom C2558
- 4 cores AMD GX-412TC
Impact of number of static routes on FreeBSD 10.2:
- 4 cores Xeon L5630
Mellanox ConnectX-4 tuning:
- Impact of queue number on forwarding performance with 2x12 cores Xeon E5-2650
Chelsio NIC T540-CR tuning:
- Impact of queue number on forwarding performance with 8 cores Xeon E5-2650
- Impact of queue number on forwarding performance with 8 cores Atom C2758
- Impact of disabling cxgbe.toecaps_allowed on forwarding performance with 8 cores Xeon E5-2650
- Impact of pinning IRQ queue to CPU with 8 cores Xeon E5-2650
- Impact of pinning IRQ queue to CPU with 8 cores Atom C2758
Intel 82599EB NIC tuning:
- Impact of Intel 82599EB queue number (1 queue per core) on forwarding performance with 4 cores Xeon L5630
- Impact of random.harvest.mask and Intel 82599EB queue number (1 queue per core) on forwarding performance with 4 cores Xeon L5630
- Impact of Intel 82599EB AIM on forwarding performance with 4 cores Xeon L5630
- Impact of Intel 82599EB Rx|Tx process limit on forwarding performance with 4 cores Xeon L5630
- Impact of Intel 82599EB descriptors per queue on forwarding performance with 4 cores Xeon L5630
- Impact of AIM on Intel 82599EB
Intel i350 tuning:
- Impact of Intel i350 number of queues(1 queue per core) on forwarding performance with 4 cores Atom C2558
- Impact of AIM on Intel i350
Packet filter (pf) tuning:
- Impact of states_hashsize table on pf performance on FreeBSD 12-head r332390 on Atom 8 core
- Impact of states_hashsize table on pf performance on FreeBSD 12-head r332390 on AMD GX
- Impact of number of flows on pf performance on FreeBSD 11.1-yandex
ipfw tuning:
- Impact of number of flows with maximum ipfw hash table on FreeBSD 11.1-yandex
IPsec performance:
- 4 cores AMD GX-412TC (PC Engines APU2C4)
- 4 cores Atom C2558 (Netgate RCC-VE 4860)
- 8 cores Atom C2758 & Chelsio T540
- 8 cores Xeon E5-2650 & Chelsio T540
- 2x12 cores Xeon E5-2650v4 & Chelsio T520
- (old) 4 cores Intel Xeon L5630 (IBM x3550-M3)
- (old) 2 cores AMD G-T40E (PC Engines APU1D)
OpenVPN performance:
- 4 cores AMD GX-412TC (PC Engines APU2C4)
- 4 cores Atom C2558 (Netgate RCC-VE 4860)
- 8 cores Atom C2758 & Chelsio T540
- 4 cores Intel Xeon L5630 (IBM x3550-M3)
- (old)2 cores AMD G-T40E (PC Engines APU1D)
- Impact of OpenVPN fast-io (PC Engines APU2C4)
Wireguard kernel vs userland:
- PC Engines APU2C4 (4 cores AMD GX-412TC @ 1GHz) and 4 Intel i210AT Gigabit NIC
- Netgate RCC-VE 4860 (4 cores Intel Atom C2558 @ 2.40GHz) and quad port Intel i350
- 8 cores Atom C2758 & Chelsio T540
Tinc performance:
- 4 cores AMD GX-412TC (PC Engines APU2C4)