Streaming parser does return refill the buffer and returns Incomplete

[lbckmnn]

Hey, thanks for the library. I tried using the sample code from the documentation on a larger pcapng file but the refilling of the buffer doesn't seem to work properly.

See the following example: pcapng file: https://wiki.wireshark.org/uploads/moin_import/attachments/SampleCaptures/iperf3-udp.pcapng.gz

minimal example

use pcap_parser::data::{get_packetdata, PacketData};
use pcap_parser::traits::PcapReaderIterator;
use pcap_parser::*;
use std::env;
use std::fs::File;

fn main() {
    let args: Vec<_> = env::args().collect();
    let file = File::open(args[1].clone()).unwrap();
    let mut reader = PcapNGReader::new(65536, file).expect("PcapNGReader");
    let mut num_packets = 0;
    loop {
        match reader.next() {
            Ok((offset, block)) => {
                match block {
                    PcapBlockOwned::NG(Block::EnhancedPacket(ref epb)) => {
                        num_packets += 1;
                    }
                    PcapBlockOwned::NG(Block::SimplePacket(ref spb)) => {
                        num_packets += 1;
                    }
                    PcapBlockOwned::NG(_) => {
                        eprintln!("unsupported block");
                    }
                    PcapBlockOwned::Legacy(_) | PcapBlockOwned::LegacyHeader(_) => unreachable!(),
                }
                reader.consume(offset);
                println!("{}", reader.position());
            }
            Err(PcapError::Eof) => break,
            Err(PcapError::Incomplete) => {
                eprintln!("Could not read complete data block.");
                eprintln!("Hint: the reader buffer size may be too small, or the input file nay be truncated.");
                break;
            }
            Err(e) => panic!("error while reading: {:?}", e),
        }
    }
    println!("num packets: {}", num_packets);
}

output

./minimal-example ./Downloads/iperf3-udp.pcapng
unsupported block
228
unsupported block
300
408
516
640
804
912
1020
1120
1256
1356
1456
1556
1660
1760
1992
2092
2192
2292
2400
2508
2632
2796
2876
2976
3056
3156
4680
4780
4880
6404
7928
9452
10976
12500
14024
15548
17072
18596
20120
21644
23168
24692
26216
27740
29264
30788
32312
0
1524
3048
4572
6096
7620
9144
10668
12192
13716
15240
16764
18288
19812
21336
22860
24384
25908
27432
28956
30480
Could not read complete data block.
Hint: the reader buffer size may be too small, or the input file nay be truncated.
num packets: 67

The reader position indicates that the buffer shifts properly when half the capacity is reached but when about the buffersize is parsed, an PcapError::Incomplete is returned. The documentation says the consume method will refill but that does not seem to be the case. If I double the buffer size I can parse about double the amount of the packets before I get an PcapError::Incomplete.

[lbckmnn]

Hey,

I just realized I need to call reader.refill() myself and found another example which actually calls the refill method when next() returns an Incomplete so this is just a documentation issue.

I found at least two places: 1.) The example here contains no refill. This is the example which is used: https://docs.rs/pcap-parser/latest/pcap_parser/struct.PcapNGReader.html#example

2.) One sentence which is probably outdated?: https://docs.rs/pcap-parser/latest/pcap_parser/traits/trait.PcapReaderIterator.html "consume takes care of refilling the buffer if required."

[evanrichter]

I just got bit by this also

[chifflier]

Hi, You're right on both points, I will fix this as soon as possible.

Better code/example is available in the pcap-analyzer project: https://github.com/rusticata/pcap-analyzer/blob/f91f83ab03ea27a512873131abbc26130b8d0e85/libpcap-tools/src/block_engine.rs#L48-L101

Note on the referenced code: when an incomplete block is encountered, you need to call reader.refill(), but you have to be careful not to create an infinite loop: if the buffer is too small, the next block may be incomplete even after refill, or if the incomplete block is at end of file. The current design is like that because it allows adjusting the buffer size dynamically

[evanrichter]

thanks for the updated docs!!