linux-flow-exporter
linux-flow-exporter copied to clipboard
wide-vsix
eBPF based IPFIX exporter. This software is an IPFIX flow-exporter for routing with Linux kernel. It records flow stats forwarded by the kernel using tc-ebpf, without AF_PACKET or conntrack. This flow...
linux-flow-exporter
eBPF based IPFIX exporter. This software is an IPFIX flow-exporter for routing with Linux kernel. It records flow stats forwarded by the kernel using tc-ebpf, without AF_PACKET or conntrack. Some flow-exporter using AF_PACKET has performance issues due to frequent user/kernel communication, and Some one using conntrack does not work properly in a multipath environment. This software is an flow-exporter that does not have such issues and supports multipath environment with less performance issues.
System Components
- ebpflow: in-kernel flow-stats collector with ebpf
- flowctl: user-space cli utility includes:
- daemonized agent for IPFIX flow-exporter
- dump the in-kernel flow-stats from the user-space
- IPFIX dummy data transmitter for test
- dependencies (tested)
- linux kernel 5.x+
- iproute2 5.18+
Requirements: you can verifiy compatibity with dependency-check subcmd.
-
clang --version: 10.0.0 -
uname -r: 5.15.0-1008-gcp -
ip -V: ip utility, iproute2-5.18.0, libbpf 0.8.0- iproute2 is needed to attach ebpf program into the kernel.
$ sudo flowctl dependency-check
clang version (expect v10.0.0): v12.0.1 (VALID)
kernel version (expect v5.4.0): v5.15.0 (VALID)
iproute2 binary version (expect v5.4.0): v5.18.0 (VALID)
iproute2 libbpf version (expect v0.8.0): v0.8.0 (VALID)
Example Usage
collectors:
- address: 10.146.0.6:2100
templates:
- id: 1024
template:
- FlowEndMilliseconds
- FlowStartMilliseconds
- OctetDeltaCount
- PacketDeltaCount
- IpVersion
- FlowDirection
- SourceIPv4Address
- DestinationIPv4Address
git clone <this-repo>
cd <this-repo>
sudo ./misc/create_netns.sh
sudo flowctl meter attach --netns ns0 -n eth1
sudo flowctl meter attach --netns ns0 -n eth2
sudo flowctl meter attach --netns ns0 -n eth3
flowctl usage
docker run --rm --name tmp -it --privileged --net=host -v /usr/include/asm:/usr/include/asm -v /var/run/netns:/var/run/netns ghcr.io/wide-vsix/linux-flow-exporter:branch-master bash
flowctl meter status
## How to check current flow cache
$ sudo flowctl dump
IFINDEX PROTO SRC DST PKTS BYTES
98 6 172.17.0.7:49375 172.67.134.3:80 1707 186818
98 6 172.17.0.7:41585 104.21.25.104:80 1710 187560
98 6 172.17.0.7:37869 104.21.25.104:80 9 486
$ sudo flowctl flush -i 98 -p 6 -s 172.17.0.7 -S 37869 -d 104.21.25.104 -D 80 # one cache
$ sudo flowctl flush --all # all caches
Limitation
This software works ONLY for tcp.
Utilities
docker run --rm -td --name tmp1 nicolaka/netshoot bash
Background Why we need
- conntrack doesn't support async traffic
- libpcap based approach consume extreamly big computing cost
REFS
- Let's read RFC regarding IPFIX (ja) by Asama-san
- SKB Definition
- Connection Tracking (conntrack): Design and Implementation Inside Linux Kernel
- Packet mark in a Cloud Native world, LPC
- VMware NSX IPFIX for Distributed Firewall
- VMware NSX IPFIX for Logical Switch
- Comparison and Practice of packet processing implementations and acceleration methods (ja), Ebiken-san, Higebu-san, JANOG45
- IN-kernel metadata propagation technique from XDP buffer to SKB
- Private Discussion for metadata practice in eBPF
- One of the Reference design for traffic control mech using eBPF(both xdp and tc), ENOG63 by Higebu-san
- An open, unofficial registry of linux packet mark bits (aka fwmark, connmark, netfilter, iptables, nftables)
- BPF Features by Linux Kernel Version
Specification
Supported Text Log Keys
supported
src, string
dst, string
proto, string
starttime
endtime
pkts
bytes
consideration to support
matched acl rule number
Supported IPFIX IETF IE
reference: IANA registration
{"FlowEndMilliseconds", netflow.IPFIX_FIELD_flowEndMilliseconds, 8},
{"FlowStartMilliseconds", netflow.IPFIX_FIELD_flowStartMilliseconds, 8},
{"FlowEndNanoseconds", netflow.IPFIX_FIELD_flowEndNanoseconds, 8},
{"FlowStartNanoseconds", netflow.IPFIX_FIELD_flowStartNanoseconds, 8},
{"OctetDeltaCount", netflow.IPFIX_FIELD_octetDeltaCount, 8},
{"PacketDeltaCount", netflow.IPFIX_FIELD_packetDeltaCount, 8},
{"IpVersion", netflow.IPFIX_FIELD_ipVersion, 1},
{"IngressInterface", netflow.IPFIX_FIELD_ingressInterface, 4},
{"EgressInterface", netflow.IPFIX_FIELD_egressInterface, 4},
{"SourceIPv4Address", netflow.IPFIX_FIELD_sourceIPv4Address, 4},
{"DestinationIPv4Address", netflow.IPFIX_FIELD_destinationIPv4Address, 4},
{"ProtocolIdentifier", netflow.IPFIX_FIELD_protocolIdentifier, 1},
{"SourceTransportPort", netflow.IPFIX_FIELD_sourceTransportPort, 2},
{"DestinationTransportPort", netflow.IPFIX_FIELD_destinationTransportPort, 2},
under the development
{"forwardingStatus", 89, 1}
follow will be supported, in mid term
{"flowDirection", netflow.IPFIX_FIELD_flowDirection, 1},
{"tcpControlBits", netflow.IPFIX_FIELD_tcpControlBits, 1},
{"icmpTypeCodeIPv4", netflow.IPFIX_FIELD_icmpTypeCodeIPv4, 2},
follow will be supported, in long term
{"ipClassOfService", netflow.IPFIX_FIELD_ipClassOfService, 1},
{"sourceIPv4PrefixLength", netflow.IPFIX_FIELD_sourceIPv4PrefixLength, 1},
{"destinationIPv4PrefixLength", netflow.IPFIX_FIELD_destinationIPv4PrefixLength, 1},
{"ipNextHopIPv4Address", netflow.IPFIX_FIELD_ipNextHopIPv4Address, 4},
{"bgpSourceAsNumber", netflow.IPFIX_FIELD_bgpSourceAsNumber, 4},
{"bgpDestinationAsNumber", netflow.IPFIX_FIELD_bgpDestinationAsNumber, 4},
{"bgpNextHopIPv4Address", netflow.IPFIX_FIELD_bgpNextHopIPv4Address, 4},
{"minimumTTL", netflow.IPFIX_FIELD_minimumTTL, 1},
{"maximumTTL", netflow.IPFIX_FIELD_maximumTTL, 1},
{"fragmentIdentification", netflow.IPFIX_FIELD_fragmentIdentification, 4},
{"vlanId", netflow.IPFIX_FIELD_vlanId, 2},
{"flowEndReason", netflow.IPFIX_FIELD_flowEndReason, 1},
{"dot1qVlanId", netflow.IPFIX_FIELD_dot1qVlanId, 2},
{"dot1qCustomerVlanId", netflow.IPFIX_FIELD_dot1qCustomerVlanId, 2},
Supported IPFIX Enterprise IE
Enterprise No: 28972 (Keio University, iana registry). It may be updated by LINE Corporation
Pre Build Packages
You can use
automated build container images or
automated build elf-binary.
Both are automatically created by updating main-branch and creating release.
Please note that flowctl depends on clang, iproute2 and linux kernel.
You can use flowctl dependency-check to check for dependency problems.
for container (latest version)
docker run -it --rm --privileged --net=host \
-v /usr/include/asm:/usr/include/asm \
-v /var/run/netns:/var/run/netns \
ghcr.io/wide-vsix/linux-flow-exporter:branch-main bash
flowctl version
for container (specified version)
docker run -it --rm --privileged --net=host \
-v /usr/include/asm:/usr/include/asm \
-v /var/run/netns:/var/run/netns \
ghcr.io/wide-vsix/linux-flow-exporter:v0.0.5 bash
flowctl version
for binary (latest version)
curl -Lo /usr/local/bin/flowctl https://github.com/wide-vsix/linux-flow-exporter/releases/download/branch-main/flowctl.linux-amd64
chmod +x /usr/local/bin/flowctl
flowctl version
for binary (specified version)
curl -Lo /usr/local/bin/flowctl https://github.com/wide-vsix/linux-flow-exporter/releases/download/v0.0.5/flowctl.linux-amd64
chmod +x /usr/local/bin/flowctl
flowctl version
Licence
The user space components are licensed under the Apache License, Version 2.0. The BPF code templates are dual-licensed under the General Public License, Version 2.0 (only) and the 2-Clause BSD License (you can use the terms of either license, at your option).
wide-vsix