New public methods for background processing

[SebMichaud]

For some programming platforms, .NET calls are possible but sub-optimized. As an example, .NET callback executions in LabVIEW are slow (let say 2ms per packet), which prevent capturing high throughput networks with OnPacketArrival (a huge temporal drift appears, but no packets are dropped).

To promote the use and diffusion of SharpPcap, which is a great lib, it would be great to have some additional methods to natively wrap the OnPacketArrival callback, almost like in the QueuingPacketsForBackgroundProcessing example.

Something like: -CreateCaptureQueue, public method to register the callback, with a LibPcapLiveDevice object as a input parameter -OnPacketArrival, private implementation of the callback, queuing packets into a List(RawCapture) object -FlushCaptureQueue, public method to flush the queue content, with a List(RawCapture) object as an output And probably other parameters to prevent memory burst.

I can build my own assembly for that (I did it, it works well), but I have the feeling that this small additions directly into SharpPcap would provide a small but great level of abstraction for people like me.

[kayoub5]

how long does it take to call a .NET function in Labview?

[kayoub5]

Is the 2ms delay accumulative? how big is the "temporal drift" ?

[SebMichaud]

Let's take an example: 10000 ICMP packets captured in 3 seconds by Wireshark. With a .NET callback in LabVIEW, it takes something like 20 seconds to get the 10000 packets, wich is huged. But if the callback is registered inside an assembly, with a queue mechanism, it takes 3 seconds to get everything into LabVIEW (FlushCaptureQueue called in a while loop every 500ms).

[kayoub5]

if you don't want callback based approach, you could still use GetNextPacket

[SebMichaud]

GetNextPacket requires a while loop for continuous capture. It delivers one packet each call and is as slow as the callback in LabVIEW. The callback is documented as performant (not in LabVIEW but in textual langages), it is used by WireSkark, and allows easy parallel processing. The only efficient method is to have the callback outside of LabVIEW.

[kayoub5]

Given that this is a LabVIEW specific problem, I don't see (yet) the benefit of introducing and maintaining such API.

You are free to contribute the code you have already developed, but I can't promise it will be accepted without seeing it first.

[SebMichaud]

I would expect the same kind of performances issues with Matlab or other specific platforms. The code is below, it's basicaly a reworked version of the example QueuingPacketsForBackgroundProcessing of SharpPcap. Keep in mind that I'm not a C# developper at all. So, I would be grateful if you give me feedbacks if you decide to not integrate the feature directly into SharpPcap.

using System;
using System.Collections.Generic;
using SharpPcap;
using SharpPcap.LibPcap;

namespace SharpPcapWrapper
{
    public class CaptureQueue
    {
        /// Object that is used to prevent concurrent writting or partial reading of PacketQueue
        private static readonly object QueueLock = new object();

        /// The queue that the callback thread puts packets in
        private static List<RawCapture> PacketQueue = new List<RawCapture>();

        /// Creation of the capture queue
        public static void CreateCaptureQueue(LibPcapLiveDevice device)
        {
            // Register our handler function to the 'packet arrival' event
            device.OnPacketArrival += new PacketArrivalEventHandler(device_OnPacketArrival);
        }

        /// The famous OnPacketArrival callback
        private static void device_OnPacketArrival(object sender, PacketCapture e)
        {
            // lock PacketQueue
            lock (QueueLock)
            {
                PacketQueue.Add(e.GetPacket());
            }
        }

        /// Checks for queued packets. If any exist it locks the QueueLock, saves a
        /// reference of the current queue for itself, puts a new queue back into
        /// place into PacketQueue and unlocks QueueLock. This is a minimal amount of
        /// work done while the queue is locked. The caller can then process queue that it saved without holding
        /// the queue lock.
        public static void FlushCaptureQueue(out List<RawCapture> CaptureQueue)
        {
            CaptureQueue = new List<RawCapture>();
            
            // lock PacketQueue
            lock (QueueLock)
            {
                if (PacketQueue.Count != 0)
                {
                    // swap queues, giving the capture callback a new one
                    CaptureQueue = PacketQueue;
                    PacketQueue = new List<RawCapture>();
                }
            }
        }

        /// Destroy the capture queue
        public static void DestroyCaptureQueue()
        {
            lock (QueueLock)
            {
                // Clear the queue
                PacketQueue.Clear();
            }
        }
    }
}

[kayoub5]

@SebMichaud Please open PR, the idea does not seems to require a lot of code, and could help with other cases such as #570

Please keep in mind that code will require few changes during the review process. For Example:

• Use BlockingCollection
• Use IDisposable

[SebMichaud]

Ok will do that 👍

[kayoub5]

@SebMichaud Check GetSequence function, it could be enough for what you need.

https://github.com/dotpcap/sharppcap/blob/beeb09734a1582b0c09f6b1e193b72df0a1166fd/SharpPcap/LibPcap/PcapDevice.cs#L503

[SebMichaud]

I already try this one, but it's too slow because because of the while loop grabbing one (GetNext)packet at a time.

[SebMichaud]

Would it be possible for a real C# developer (who I'm not) to grab my PR and fix it ? My PR doesn't work and I don't know why. The BlockingCollection needs to be cast to a simple List in the Flush method, because it's way more convenient for the final user. Any help ?