tcpflow icon indicating copy to clipboard operation
tcpflow copied to clipboard
verified publisher icon simsong

unceasing flow recording

Open vadka opened this issue 11 years ago • 14 comments

Hello, tcpflow version: 1.4.4 system: gentoo (libpcap 1.3.0), ubuntu (libpcap 1.5.3)

We use tcpflow to parse TCP session from pcap files. tcpdump cmd: tcpdump -s0 -G120 ... tcp port 80 or port 1935 tcpflow cmd: tcpflow -a -Fk -o /path -r /file2 -R /file1

We got unusual behaviour of tcpflow. If both of pcap files include the same established session (like online tv), tcpflow will endlessly write the flow to file until user interrupts it. The size of such pcap files should be greater then 10 MB.The '-b' option doesn't resolve this issue too.

If you execute tcpflow without '-R' option or dump size less then 10MB, flow will be saved correctly. I can send such example if it will help to you.

vadka avatar Apr 07 '14 12:04 vadka

Thanks for the bug report. Can you provide me with a set of PCAP files that will produce the result?

On Apr 7, 2014, at 8:45 AM, vadka [email protected] wrote:

Hello, tcpflow version: 1.4.4 system: gentoo (libpcap 1.3.0), ubuntu (libpcap 1.5.3)

We use tcpflow to parse TCP session from pcap files. tcpdump cmd: tcpdump -s0 -G120 ... tcp port 80 or port 1935 tcpflow cmd: tcpflow -a -Fk -o /path -r /file2 -R /file1

We got unusual behaviour of tcpflow. If both of pcap files include the same established session (like online tv), tcpflow will endlessly write the flow to file until user interrupts it. The size of such pcap files should be greater then 10 MB.The '-b' option doesn't resolve this issue too.

If you execute tcpflow without '-R' option or dump size less then 10MB, flow will be saved correctly. I can send such example if it will help to you.

— Reply to this email directly or view it on GitHub.

simsong avatar Apr 07 '14 13:04 simsong

Sure, but it's near 20MB. Is it normal to attach files here or it's better to send by email?

vadka avatar Apr 07 '14 14:04 vadka

You can find files here: http://monitor.adminso.ru/test/test.tar.gz

tcpflow -a -q -Fk -b 10485760 -o ./ -r ./dump_1396861296 -R ./dump_1396861176 Check file 000/091.206.048.075.01935-192.168.075.006.37347

vadka avatar Apr 07 '14 15:04 vadka

Thanks. I am downloading the files now and will have an answer for you within a few days.

On Apr 7, 2014, at 11:05 AM, vadka [email protected] wrote:

You can find files here: http://monitor.adminso.ru/test/test.tar.gz

tcpflow -a -q -Fk -b 10485760 -o ./ -r ./dump_1396861296 -R ./dump_1396861176 Check file 000/091.206.048.075.01935-192.168.075.006.37347

— Reply to this email directly or view it on GitHub.

simsong avatar Apr 07 '14 16:04 simsong

Was there ever any action taken on this? I might be seeing a similar issue. We get small pcap files that explode into many gigabyte files, mostly filled with null characters. Wireshark seems to handle the files ok, and shows the session length as expected. Will try to gather some data if possible. Thanks.

randysch avatar Apr 07 '15 18:04 randysch

Which version are you using?

On Tue, Apr 7, 2015 at 2:21 PM, randysch [email protected] wrote:

Was there ever any action taken on this? I might be seeing a similar issue. We get small pcap files that explode into many gigabyte files, mostly filled with null characters. Wireshark seems to handle the files ok, and shows the session length as expected. Will try to gather some data if possible. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90688501.

simsong avatar Apr 07 '15 18:04 simsong

1.4.4, built from source.

On 04/07/2015 02:29 PM, Simson L. Garfinkel wrote:

Which version are you using?

On Tue, Apr 7, 2015 at 2:21 PM, randysch [email protected] wrote:

Was there ever any action taken on this? I might be seeing a similar issue. We get small pcap files that explode into many gigabyte files, mostly filled with null characters. Wireshark seems to handle the files ok, and shows the session length as expected. Will try to gather some data if possible. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90688501.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90690778.

randysch avatar Apr 07 '15 19:04 randysch

Well, there are two general problems here. The first is what do you do when you have a large gap in the sequence number. When do you decide to create a new connection, and when do you decide to pad? There is a tcpflow option (-m) that allows you to control this. Try running "-m 100000" and see what it does.

On Tue, Apr 7, 2015 at 3:10 PM, randysch [email protected] wrote:

1.4.4, built from source.

On 04/07/2015 02:29 PM, Simson L. Garfinkel wrote:

Which version are you using?

On Tue, Apr 7, 2015 at 2:21 PM, randysch [email protected] wrote:

Was there ever any action taken on this? I might be seeing a similar issue. We get small pcap files that explode into many gigabyte files, mostly filled with null characters. Wireshark seems to handle the files ok, and shows the session length as expected. Will try to gather some data if possible. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90688501.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90690778.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-90702510.

simsong avatar Apr 07 '15 19:04 simsong

That option had no effect. Also tried using "-b xxx" which reads like it might have an impact, but again no change.

randysch avatar Apr 08 '15 18:04 randysch

I recently stumbled across what might be the same problem or at least related. I have isolated the issue in my case to a flow which is "double" captured in the flow. The same ip:port<->ip:port appear in the trace file co-located in time but with different MAC addresses and different sequence numbers. They are logically the same flow.

Some sanitize data as an example from a tcpdump output:

    1 22:34:51.531823 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [S], seq 2645014300, win 8192, options [mss 1350,nop,wscale 2,nop,nop,sackOK], length 0
    2 22:34:51.531985 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [S], seq 3984945811, win 8192, options [mss 1350,nop,wscale 2,nop,nop,sackOK], length 0
    3 22:34:51.858379 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514216, win 16537, length 0
    4 22:34:51.858389 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333508, win 16537, length 0
    5 22:34:58.341442 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014301:2645014321, ack 694514242, win 16531, length 20
    6 22:34:58.341649 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945812:3984945832, ack 3822333534, win 16531, length 20
    7 22:35:05.550171 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014321:2645014335, ack 694514262, win 16526, length 14
    8 22:35:05.550177 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945832:3984945846, ack 3822333554, win 16526, length 14
    9 22:35:06.413529 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514280, win 16521, length 0
   10 22:35:06.413540 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333572, win 16521, length 0
   11 22:35:06.509601 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014335:2645014374, ack 694514280, win 16521, length 39
   12 22:35:06.509774 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945846:3984945885, ack 3822333572, win 16521, length 39
   13 22:35:07.171162 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514328, win 16509, length 0
   14 22:35:07.171351 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333620, win 16509, length 0
   15 22:35:07.333200 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014374:2645014380, ack 694514328, win 16509, length 6
   16 22:35:07.333206 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945885:3984945891, ack 3822333620, win 16509, length 6
   17 22:35:08.952465 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014374:2645014380, ack 694514328, win 16509, length 6
   18 22:35:08.952508 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945885:3984945891, ack 3822333620, win 16509, length 6
   19 22:35:10.233283 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014380:2645014965, ack 694514342, win 16506, length 585
   20 22:35:10.233295 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945891:3984946476, ack 3822333634, win 16506, length 585
   21 22:35:10.759340 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014965:2645014971, ack 694514378, win 16497, length 6
   22 22:35:10.759350 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984946476:3984946482, ack 3822333670, win 16497, length 6
   23 22:35:11.429804 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514399, win 16492, length 0
   24 22:35:11.429940 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333691, win 16492, length 0
   25 22:35:11.473316 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [R.], seq 2645014971, ack 694514399, win 0, length 0
   26 22:35:11.473331 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [R.], seq 3984946482, ack 3822333691, win 0, length 0

tcpflow for this sample data generated two multigigabyte (sparse) flow files. In an instance where I was working with a much larger pcap (~3GB) that exhibited similar issues tcpflow to get hung up and did not complete after several hours or processing.

My workaround after discovering the issue was to use BPF to ignore one of the "conversations". However I'm interested in seeing if tcpflow itself could handle the situation better as in the instance of new data that needs to be analyzed I basically have to discover the unique BPF for that data set in order to process it.

I'm only now looking into the code but I thought the above "example" might help shed some light onto the situation.

I unfortunately can not share the raw pcap although if I can find a small enough sample I may be able to sanitize it enough to get sharing approved.

FusionFC avatar Aug 20 '15 19:08 FusionFC

Pretty neat.

Why don't you run it with debugging turned on? tcpflow ignores MAC addresses. This should just look like retransmitted packets. Are you sure that the packets aren't being modified in route?

On Aug 20, 2015, at 3:15 PM, FusionFC [email protected] wrote:

I recently stumbled across what might be the same problem or at least related. I have isolated the issue in my case to a flow which is "double" captured in the flow. The same ip:port<->ip:port appear in the trace file co-located in time but with different MAC addresses and different sequence numbers. They are logically the same flow.

Some sanitize data as an example from a tcpdump output:

1 22:34:51.531823 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [S], seq 2645014300, win 8192, options [mss 1350,nop,wscale 2,nop,nop,sackOK], length 0
2 22:34:51.531985 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [S], seq 3984945811, win 8192, options [mss 1350,nop,wscale 2,nop,nop,sackOK], length 0
3 22:34:51.858379 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514216, win 16537, length 0
4 22:34:51.858389 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333508, win 16537, length 0
5 22:34:58.341442 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014301:2645014321, ack 694514242, win 16531, length 20
6 22:34:58.341649 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945812:3984945832, ack 3822333534, win 16531, length 20
7 22:35:05.550171 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014321:2645014335, ack 694514262, win 16526, length 14
8 22:35:05.550177 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945832:3984945846, ack 3822333554, win 16526, length 14
9 22:35:06.413529 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514280, win 16521, length 0

10 22:35:06.413540 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333572, win 16521, length 0 11 22:35:06.509601 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014335:2645014374, ack 694514280, win 16521, length 39 12 22:35:06.509774 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945846:3984945885, ack 3822333572, win 16521, length 39 13 22:35:07.171162 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514328, win 16509, length 0 14 22:35:07.171351 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333620, win 16509, length 0 15 22:35:07.333200 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014374:2645014380, ack 694514328, win 16509, length 6 16 22:35:07.333206 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945885:3984945891, ack 3822333620, win 16509, length 6 17 22:35:08.952465 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014374:2645014380, ack 694514328, win 16509, length 6 18 22:35:08.952508 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945885:3984945891, ack 3822333620, win 16509, length 6 19 22:35:10.233283 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014380:2645014965, ack 694514342, win 16506, length 585 20 22:35:10.233295 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984945891:3984946476, ack 3822333634, win 16506, length 585 21 22:35:10.759340 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 2645014965:2645014971, ack 694514378, win 16497, length 6 22 22:35:10.759350 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [P.], seq 3984946476:3984946482, ack 3822333670, win 16497, length 6 23 22:35:11.429804 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 694514399, win 16492, length 0 24 22:35:11.429940 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [.], ack 3822333691, win 16492, length 0 25 22:35:11.473316 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [R.], seq 2645014971, ack 694514399, win 0, length 0 26 22:35:11.473331 IP 192.168.1.31.YY > 192.168.2.12.XX: Flags [R.], seq 3984946482, ack 3822333691, win 0, length 0

tcpflow for this sample data generated two multigigabyte (sparse) flow files. In an instance where I was working with a much larger pcap (~3GB) that exhibited similar issues tcpflow to get hung up and did not complete after several hours or processing.

My workaround after discovering the issue was to use BPF to ignore one of the "conversations". However I'm interested in seeing if tcpflow itself could handle the situation better as in the instance of new data that needs to be analyzed I basically have to discover the unique BPF for that data set in order to process it.

I'm only now looking into the code but I thought the above "example" might help shed some light onto the situation.

I unfortunately can not share the raw pcap although if I can find a small enough sample I may be able to sanitize it enough to get sharing approved.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-133128452.

simsong avatar Aug 20 '15 23:08 simsong

I was given pcap data to analyze and in this case do not have visibility into the network and tapping infrastructure from which it comes. It would seem possible that there is a device modifying the packets somewhere in route and that I'm getting a copy from upstream and down stream of that modification.

I was looking only at the data segments themselves and the data has not been modified. I can isolate the flows using bpf and extract identical flows. I'm just now digging deeper into the raw data.

./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: TCPFLOW version 1.4.4 
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: looking for handler for datalink type 1 for interface sample.pcap
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952995
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021484
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952995
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021484
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952975
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021464
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952961
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021450
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952922
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021411
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952922
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021411
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952916
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310021405
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-1649952331
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--520,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--520: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: new flow flow[192.168.1.31:YY->192.168.2.12:XX]. path:  next seq num (nsn):-310020820
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: retrying_open ::open(fn=192.168.001.031.YYYYY-192.168.002.012:XXXXX--510,oflag=x42,mask:x1b6)=6
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: created new file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: 192.168.001.031.YYYYY-192.168.002.012:XXXXX--510: closing file in tcpip::close_file
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: process_pkt..............................................................................
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: Open FDs at end of processing:      0
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: demux.max_open_flows:               1
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: Flow map size at end of processing: 0
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: Flows seen:                         16
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: Total flows processed: 16
./BUILD/tcpflow/tcpflow-1.4.4/src/tcpflow: Total packets processed: 16

FusionFC avatar Aug 24 '15 16:08 FusionFC

I've pretty much got a handle on the issues I was facing. The data itself was going to be problematic as there does appear to a device rewriting packets. However, the unceasing portion was due to a few issues with tcpflow, some of which were fixed in the git repo already. When cloned the repo and ran the latest code the behavior was better. I tracked that down mostly to a missing O_EXCL in tcpip::open_flow.

I was compiling my code from http://www.digitalcorpora.org/downloads/tcpflow/ partly because github can be a bit problematic to access due to work policies. If you are planning to continue publishing the packages there I recommend you get the latest 1.4.5 release up there.

My processing of my small test cases "works" with 1.4.5 but I still see some abnormally large flow files (1.2G flow generated from a 6.2K pcap file). I'm going to open a different issue on that.

FusionFC avatar Sep 08 '15 15:09 FusionFC

Glad you are getting the hang of it. Let me know when you figure out the 1.2G and submit a pull request, then we will issue a new release.

On Sep 8, 2015, at 11:06 AM, FusionFC [email protected] wrote:

I've pretty much got a handle on the issues I was facing. The data itself was going to be problematic as there does appear to a device rewriting packets. However, the unceasing portion was due to a few issues with tcpflow, some of which were fixed in the git repo already. When cloned the repo and ran the latest code the behavior was better. I tracked that down mostly to a missing O_EXCL in tcpip::open_flow.

I was compiling my code from http://www.digitalcorpora.org/downloads/tcpflow/ http://www.digitalcorpora.org/downloads/tcpflow/ partly because github can be a bit problematic to access due to work policies. If you are planning to continue publishing the packages there I recommend you get the latest 1.4.5 release up there.

My processing of my small test cases "works" with 1.4.5 but I still see some abnormally large flow files (1.2G flow generated from a 6.2K pcap file). I'm going to open a different issue on that.

— Reply to this email directly or view it on GitHub https://github.com/simsong/tcpflow/issues/73#issuecomment-138594787.

simsong avatar Sep 09 '15 01:09 simsong