netsampler
Procedures to decode hsflowd along with sflow v5
Hi! Great resource @lspgn, thanks for taking goflow to the moon!.. tried and liked it very much, very high performance with minimum resource use... Only problem i have right now is the hsflowd packets. What should be the way to update the goflow2 to be able decode and insert new sflow types all in together to the same db in one machine. e.g.; hsflowd + sflow v5 ---> goflow2 ---> kafka --> ch
flow_data | 0 | 2100 | extended_socket_ipv4 | sFlow Host Structures
flow_data | 0 | 2101 | extended_socket_ipv6 | sFlow Host Structures
+
sample_data | 0 | 1 | flow_sample | sFlow Version 5
sample_data | 0 | 2 | counter_sample | sFlow Version 5
sample_data | 0 | 3 | flow_sample_expanded | sFlow Version 5
sample_data | 0 | 4 | counter_sample_expanded | sFlow Version 5
flow_data | 0 | 1 | sampled_header | sFlow Version 5
flow_data | 0 | 2 | sampled_ethernet | sFlow Version 5
flow_data | 0 | 3 | sampled_ipv4 | sFlow Version 5
flow_data | 0 | 4 | sampled_ipv6 | sFlow Version 5
flow_data | 0 | 1001 | extended_switch | sFlow Version 5
flow_data | 0 | 1002 | extended_router | sFlow Version 5
flow_data | 0 | 1003 | extended_gateway | sFlow Version 5
flow_data | 0 | 1004 | extended_user | sFlow Version 5
flow_data | 0 | 1005 | extended_url (deprecated) | sFlow Version 5
flow_data | 0 | 1006 | extended_mpls | sFlow Version 5
flow_data | 0 | 1007 | extended_nat | sFlow Version 5
flow_data | 0 | 1008 | extended_mpls_tunnel | sFlow Version 5
flow_data | 0 | 1009 | extended_mpls_vc | sFlow Version 5
flow_data | 0 | 1010 | extended_mpls_FTN | sFlow Version 5
flow_data | 0 | 1011 | extended_mpls_LDP_FEC | sFlow Version 5
flow_data | 0 | 1012 | extended_vlantunnel | sFlow Version 5
or another combination..
Sincerely, F.
Hello @mrstellion, Thank you for the kind feedback.
At the moment, it is not possible to fetch the others sFlow types. I am currently evaluating if this could be done with a configuration file. You would need to edit the code of the decoder: https://github.com/netsampler/goflow2/blob/17a96d991149c9bcc1481795de1c19b718163bb3/decoders/sflow/sflow.go#L132 And then to map it into protobuf. https://github.com/netsampler/goflow2/blob/17a96d991149c9bcc1481795de1c19b718163bb3/producer/producer_sf.go#L248-L257
Let me know if this answers your question
This said, do you have a link to the spec of the new structures?
Thank you for the fast response @lspgn! That would be awesome to implement such selection process via configuration file. example sflowtool output for hsflowd looks like this;
startDatagram =================================
datagramSourceIP 10.0.0.160
datagramSize 1332
unixSecondsUTC 1402004767
datagramVersion 5
agentSubId 100000
agent 10.0.0.233
packetSequenceNo 340132
sysUpTime 17479000
samplesInPacket 7
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 876
sourceId 2:1
counterBlock_tag 0:2001
adaptor_0_ifIndex 2
adaptor_0_MACs 1
adaptor_0_MAC_0 6c641a000459
counterBlock_tag 0:2005
disk_total 0
disk_free 0
disk_partition_max_used 0.00
disk_reads 980
disk_bytes_read 4014080
disk_read_time 1501
disk_writes 0
disk_bytes_written 0
disk_write_time 0
counterBlock_tag 0:2004
mem_total 2056589312
mem_free 1100533760
mem_shared 0
mem_buffers 33464320
mem_cached 807546880
swap_total 0
swap_free 0
page_in 35947
page_out 0
swap_in 0
swap_out 0
counterBlock_tag 0:2003
cpu_load_one 0.390
cpu_load_five 0.440
cpu_load_fifteen 0.430
cpu_proc_run 1
cpu_proc_total 95
cpu_num 2
cpu_speed 0
cpu_uptime 770774
cpu_user 160600160
cpu_nice 192970
cpu_system 77855100
cpu_idle 1302586110
cpu_wio 4650
cpuintr 0
cpu_sintr 308370
cpuinterrupts 1851322098
cpu_contexts 800650455
counterBlock_tag 0:2006
nio_bytes_in 405248572711
nio_pkts_in 394079084
nio_errs_in 0
nio_drops_in 0
nio_bytes_out 406139719695
nio_pkts_out 394667262
nio_errs_out 0
nio_drops_out 0
counterBlock_tag 0:2000
hostname cumulus
UUID fd-01-78-45-93-93-42-03-a0-5a-a3-d7-42-ac-3c-de
machine_type 7
os_name 2
os_release 3.2.46-1+deb7u1+cl2+1
endSample ----------------------
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 876
sourceId 0:44
counterBlock_tag 0:1005
ifName swp42
counterBlock_tag 0:1
ifIndex 44
networkType 6
ifSpeed 0
ifDirection 2
ifStatus 0
ifInOctets 0
ifInUcastPkts 0
ifInMulticastPkts 0
ifInBroadcastPkts 0
ifInDiscards 0
ifInErrors 0
ifInUnknownProtos 4294967295
ifOutOctets 0
ifOutUcastPkts 0
ifOutMulticastPkts 0
ifOutBroadcastPkts 0
ifOutDiscards 0
ifOutErrors 0
ifPromiscuousMode 0
endSample ----------------------
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 1022129
sourceId 0:7
meanSkipCount 128
samplePool 130832512
dropEvents 0
inputPort 7
outputPort 10
flowBlock_tag 0:1
flowSampleType HEADER
headerProtocol 1
sampledPacketSize 1518
strippedBytes 4
headerLen 128
headerBytes 6C-64-1A-00-04-5E-E8-E7-32-77-E2-B5-08-00-45-00-05-DC-63-06-40-00-40-06-9E-21-0A-64-0A-97-0A-64-14-96-9A-6D-13-89-4A-0C-4A-42-EA-3C-14-B5-80-10-00-2E-AB-45-00-00-01-01-08-0A-5D-B2-EB-A5-15-ED-48-B7-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35-36-37-38-39-30-31-32-33-34-35
dstMAC 6c641a00045e
srcMAC e8e73277e2b5
IPSize 1500
ip.tot_len 1500
srcIP 10.100.10.151
dstIP 10.100.20.150
IPProtocol 6
IPTOS 0
IPTTL 64
TCPSrcPort 39533
TCPDstPort 5001
TCPFlags 16
endSample ----------------------
I am checking this three;
https://sflow.org/developers/structures.php
https://sflow.org/developers/specifications.php
https://github.com/google/gopacket/blob/master/layers/sflow.go
to find out more about the list of the standard, sflow-defined structures.
Secondary question on my mind is; Does the collector needs to be seperated or can it be collected together with the same pipe since they are just counter and flow samples with different types ? e.g.:
agent 1 hsflowd ----->.1:6343
{ [GOFLOW2] - Kafka - CH } *One Collector*
agent 2 [sflowv5] ----->.1:6343
Finally, may i request partial / complete hsflowd decode feature at least as an example to evaluate and proceed to produce more?
Sincerely, F.
Hello,
That would be awesome to implement such selection process via configuration file.
I agree, this would also simplify #12 and #11. Currently, only NetFlow/IPFIX allows custom mapping of their fields and packet bytes extraction by offset and length.
Thank you for sharing the links and the example.
Does the collector needs to be seperated or can it be collected together with the same pipe since they are just counter and flow samples with different types ?
I believe hsflowd samples are sFlow, so there should not be any issues having a single collector but for now, only specific fields will be extracted. Unless I am misunderstanding your question?
Finally, may i request partial / complete hsflowd decode feature at least as an example to evaluate and proceed to produce more?
I will try to find some time to work on a configuration based extraction. But I will not create a decoder for those specific fields for now.
Started thinking on how to do it (branch feature/custom-map-sflow).
https://github.com/netsampler/goflow2/blob/cb238e17d1ba78cfea76e7c13d9f08a051541dec/decoders/sflow/sflow.go#L251-L253
I realize that custom decoding of certain structures (non fixed length) might be tricky with single offset+length configuration. Might be able to play around with reflect and how to explore structures (eg: map extended_nat.address into CustomBytes1).