tcpflow
tcpflow copied to clipboard
simsong
"Zero-length" frames result in incorrectly extracted streams
Found a situation that appears to prevent tcpflow from properly extracting streams.
In some traffic, the IP Length field (ip.len in Wireshark/tshark) is zero. WS/ts assumes this is because of TCP segmentation offloading, per its UI. Output from tcpflow is the correct length, but is populated with null bytes. In my testing, this appeared to occur only with frames larger than 1500 bytes.
I'm trying to collect a sample pcap file that demonstrates this, but the one I used is customer-based and I cannot release it. Trying to reproduce data to reflect this, but I can run any test code against this data in the mean time.
Can you give me an RFC reference for this TCP behavior?
Thanks
Sent from my phone.
On Jan 3, 2017, at 8:27 AM, Phil Hagen [email protected] wrote:
Found a situation that appears to prevent tcpflow from properly extracting streams.
In some traffic, the IP Length field (ip.len in Wireshark/tshark) is zero. WS/ts assumes this is because of TCP segmentation offloading, per its UI. Output from tcpflow is the correct length, but is populated with null bytes. In my testing, this appeared to occur only with frames larger than 1500 bytes.
I'm trying to collect a sample pcap file that demonstrates this, but the one I used is customer-based and I cannot release it. Trying to reproduce data to reflect this, but I can run any test code against this data in the mean time.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
I'm not sure there is a final RFC, but VMware does this quite commonly. There are other, more generic offloads (Large-Segment Offload by Windows), but I don't know whether those approaches result in the same "zero-length" artifacts in the ip.len field.
Some resources that detail this behavior: https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html
The closest thing I can find to an RFC is here: https://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01
Okay. I don’t think that I can fix this without packets, but if you submit a fix, I’m happy to look at it!
On Jan 4, 2017, at 10:15 AM, Phil Hagen [email protected] wrote:
I'm not sure there is a final RFC, but VMware does this quite commonly. There are other, more generic offloads (Large-Segment Offload by Windows), but I don't know whether those approaches result in the same "zero-length" artifacts in the ip.len field.
Some resources that detail this behavior: https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://ask.wireshark.org/questions/16279/why-are-the-bytes-00-00-but-wireshark-shows-an-ip-total-length-of-2016 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055140 https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-E105A601-9331-496C-A213-F76EA3863E31.html — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142#issuecomment-270394702, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhTrP0RVStzhmjfUT-2g7cSlPAbweFaks5rO7edgaJpZM4LZnel.
Roger that and thanks - working to collect those. May involve some redaction, but I think we can get something to demonstrate this...
The colleague who brought this to my attention has a pcap that I can share but not publicly - is your work email OK?
Please use my gmail address.
On Jan 5, 2017, at 12:31 PM, Phil Hagen [email protected] wrote:
The colleague who brought this to my attention has a pcap that I can share but not publicly - is your work email OK?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142#issuecomment-270704625, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhTrICsaf5ue0pH6-T8CBjBS6OX826sks5rPSjZgaJpZM4LZnel.
- I’ve included [redacted].pcap in the zip file. (sha256 b28393de3a3ec2316adced575178e200e970846f2762c9e003a5185b827ea5ac). This sample is from a colleague who permitted me to share it with you, but it cannot be shared publicly.
- Ran tcpflow 1.4.4, 1.4.5, and the current checkout of 1.4.6 against the pcap file, with identical results (see sha256sums, included in attached zip)
- loaded pcap into wireshark (v2.0.4, but have previously tested with additional versions)
- in wireshark, frame 10 has “ip.len" field consisting of zero-bytes. Wireshark states "Total Length: 2276 bytes (reported as 0, presumed to be because of "TCP segmentation offload" (TSO))”.
- saved equivalent raw tcp stream from wireshark to a file ("[redacted].wireshark”, included in attached zip file)
- Examined tcpflow output file "[redacted]”, which corresponds with the TCP stream for frame 10. From offset 0x0653-0x0f0f (2236 bytes), null bytes exist in the tcpflow file but not the stream extracted from wireshark. This segment size is equal to the wireshark tcp.len field in frame 10 - not sure if this is relevant or coincidental.
From my research it appears this only happens with frames that are larger than 1500 bytes. This is likely the result of VMware’s TSO functionality, but as I don’t have direct access to this infrastructure, I cannot 100% confirm that’s the case.
Hi, we are having same issue: the third segment has length zero, caused by TSO. We disabled TSO and problem is solved. NIC is Intel Corporation 82599ES 10-Gigabit. Disabling TSO is just workaround while waiting for final solution. TSO is supposed to decrease latency, but in our case it was increasing latency.
What’s TSO?
On Dec 15, 2019, at 4:16 AM, alaertegv [email protected] wrote:
Hi, we are having same issue: the third segment has length zero, caused by TSO. We disabled TSO and problem is solved. NIC is Intel Corporation 82599ES 10-Gigabit. Disabling TSO is just workaround while waiting for final solution. TSO is supposed to decrease latency, but in our case it was increasing latency.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/simsong/tcpflow/issues/142?email_source=notifications&email_token=AAMFHLHJIGRPBSH7VWQHJ2LQYXYWNA5CNFSM4C3GO6S2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG4UVAY#issuecomment-565791363, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMFHLCWY3HCI7GAZBXD653QYXYWNANCNFSM4C3GO6SQ.
TCP Segmentation Offloading - where the segmentation is handled by hardware on the NIC rather than the OS stack.