reading gunzip pcap.

[zolt4n]

Do you think of implementing reading pcap.gz and pcap.zst file format ?

[seladb]

the zstd format is already available for pcapng files: https://pcapplusplus.github.io/docs/features#read-and-write-packets-fromto-files

The reason it's not available for pcap files is because PcapPlusPlus uses libpcap/WinPcap/Npcap under the hood to read and write pcap files, and as far as I know these libraries don't support gz and zstd.

You may ask how is it implemented for pcapng? Well, this is because for PcapPlusPlus uses LightPcapNg for parsing pcapng files, and zstd support was added there

[zolt4n]

I think there is a way to do it with libpcap too. We would need to have something reading from a gz/zst file to a pipe and libpacp reading from that pipe. That would need to be multithreaded though. (at least I think)

[seladb]

yes, that might work, but will require PcapPlusPlus to have Zlib and libzstd as dependencies. For libzstd it's less of a problem because it's already there, but we'll need to add Zlib as well.

Also opening a separate thread is not optimal... why can't it be on the same thread?

[zolt4n]

Adding zlib is not a burden is it ?

Hum yeah it's not required you can have a non blocking read to the file and write the result directly to the pipe that would work I guess.

[seladb]

I'm not sure how this would work with large files: libpcap's pcap_open_offline() gets a file path as an argument. That means that we first need to unpack the archived file and save it to a temp folder, and then give the path to libpcap. That might be ok for small files, but what about large files? 🤔

UPDATE: I just realized that libpcap exposes a pcap_fopen_offline() method that takes a FILE* instead of a path. I think WinPcap and Npcap has it also, so this may solve the large files issue

Anyway, adding zlib as an optional dependency (like libzstd) is indeed possible, but would require quite a lot of effort:

• Add zlib to PcapPlusPlus build system:
  • Add PcapPlusPlus.mk.zlib that includes the zlib dependency and use it in all of the configuration scripts (configure-*)
  • Adding zlib to Visual Studio template projects
• Write the code that can take a compressed pcap/pcapng file, open it and feed it to libpcap (assuming we solve the large file problem somehow)
• What about writing to compressed files?
• Writing tests for all of that
• Add this into PcapPlusPlus CI

[zolt4n]

Writing of compressed file would be a plus as well for the library. But I am personally more interested into the pcapng writing capability.

[seladb]

pcapng already supports writing to zstd format: https://pcapplusplus.github.io/docs/features#read-and-write-packets-fromto-files

To use it you need to build PcapPlusPlus with a special flag, as described for the different platforms: https://pcapplusplus.github.io/docs/install#build-from-source

[seladb]

@zolt4n please let me know how we should proceed?

[zolt4n]

I won't have time to work on that before several month I think.

[seladb]

@zolt4n from your previous answer it seems that you're interested in pcapng compression. If that is the case, zstd compression is already supported. Will this work for your use-case?

[rajnishdahiya]

@seladb A small but related issue. Lot of times I see .zst as file extension rather than .zstd. I can create a PR if you like, but it's literally one line totally innocent change in light_zstd_compression.c if you just want to push it in master directly.

$ git diff
diff --git a/3rdParty/LightPcapNg/LightPcapNg/src/light_zstd_compression.c b/3rdParty/LightPcapNg/LightPcapNg/src/light_zstd_compression.c
index a2e7bf4..4423983 100644
--- a/3rdParty/LightPcapNg/LightPcapNg/src/light_zstd_compression.c
+++ b/3rdParty/LightPcapNg/LightPcapNg/src/light_zstd_compression.c
@@ -110,7 +110,7 @@ void free_zstd_decompression_context(_decompression_t* context)

 int is_zstd_compressed_file(const char* file_path)
 {
-       if (strstr(file_path, ".zstd"))
+       if (strstr(file_path, ".zstd") || strstr(file_path, ".zst"))
        {
                return 1;
        }

[seladb]

@seladb A small but related issue. Lot of times I see .zst as file extension rather than .zstd. I can create a PR if you like, but it's literally one line totally innocent change in light_zstd_compression.c if you just want to push it in master directly.

Thank you @rajnishdahiya for bringing this up! Actually the change should happen in 2 places:

• In light_zstd_compression.c as you mentioned
• In PcapFileDevice.cpp

Would you consider opening a small PR for this fix (and add a quick test)?

[rajnishdahiya]

okay, I will create a PR in couple of weeks.

[seladb]

@rajnishdahiya are you still considering opening a PR with this small fix?

[rajnishdahiya]

Hello @seladb

Actually I had shifted to another project, even before Pcap++ was integrated into the previous project. That project is on hold and might get resumed in near future but no timeline. In short, I can still make the above change when I start using pcap++ but no visibility on timeline.

Thanks Rajnish Mb: +91-9818398146

On Mon, Aug 1, 2022 at 4:03 AM seladb @.***> wrote:

@rajnishdahiya https://github.com/rajnishdahiya are you still considering opening a PR with this small fix?

— Reply to this email directly, view it on GitHub https://github.com/seladb/PcapPlusPlus/issues/731#issuecomment-1200514438, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXPQ2EJZWJBWHQALVYY55LVW35KZANCNFSM5F3PEUMA . You are receiving this because you were mentioned.Message ID: @.***>

[seladb]

Thanks for letting me know @rajnishdahiya ! We can keep this issue open for now until you or someone else can work on this

[seladb]

Fixed in #978