Error: 0xe1000014 The buffer was incompletely grabbed

[kesaroid]

Describe what you want to implement and what the issue & the steps to reproduce it are:

I have a setup with 14 Basler a2A1920-51gcPRO GigE cameras all required to run at 24 fps. Cat6A cables are used that are maximum 10 meters in length (some shorter) leading to a patch panel and then to a network switch. All 14 Basler cameras are connected to the same network switch which is then connected to the server. As soon as I open more than a 3-4 cameras, we start noticing dropped frames which subside over time. However, if all cameras are open, the dropped frames are continuous.

Below image is just to show when all cameras are running, the network uses about 6.7Gbps/10Gbps network bandwidth and dropped frames still persist (Ignore camera parameters)

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

Hardware setup: Server: DELL XE2420 CPU architecture: x64 OS: Windows RAM: 32GB

Interface: Camera model: Basler a2A1920-51gcPRO Network switch: NetGear S3300 28+ Network adapter: Intel X710-T2L NIC Cable type/Length: CAT 6A/Maximum 10 meters

Network adapter allows Jumbo packets, some important adapter settings: Interrupt moderation: Enabled Interrupt moderation rate: Low Jumbo packets: 9014 bytes (max) Receive buffers: 4096 (max)

The ideal transport layer parameters suggested by Bandwidth manager is as follows:

Packet size: 8192 (max) Inter-packet delay: 37035

Also, Bandwidth reserve mode is set to Performance

Required camera settings: Acquisition rate: 24 Exposure time: 10000 Width, Height: 1920, 1080

Runtime information:

python: 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)]
platform: win32/AMD64/10
pypylon: 1.9.0 / 6.3.0.18933

Full error: 
Basler a2A1920-51gcPRO (40113545)	Image acquisition on "Basler a2A1920-51gcPRO (40113545)" failed! Error: 0xe1000014 "The buffer was incompletely grabbed. This can be caused by performance problems of the network hardware used, i.e. network adapter, switch, or ethernet cable. To fix this, try increasing the camera's Inter-Packet Delay in the Transport Layer category to reduce the required bandwidth, and adjust the camera's Packet Size setting to the highest supported frame size."

[thiesmoeller]

Is the flow control in your switch set to symmetric?

[SMA2016a]

do you use both uplink ports? Since each camera needs just 55MB/sec set the bandwidth on 60% for each camera in bandwidth manager. try to build up 3 grupe of cameras using frame transmission delay parameters, instead of using a high value for interpacket delay.

G1 = interpacket delay 10000 image transmission delay 0 G2 = interpacket delay 10000 image transmission delay 9000 G3 = interpacket delay 10000 image transmission delay 18000

[kesaroid]

@thiesmoeller by default the uplink port's flow control was disabled. But I was able to change it to symmetric but saw no benefits. So it is not resolved yet.

P.S I have set Bandwidth Reserve mode to Performance (Bandwidth reserve: 3, Bandwidth reserve accumalation:10)

[kesaroid]

@SMA2016a I have set the bandwidth to 60% i.e 75Mbps, but that automatically increases the inter-packet delay to 37035. I don't think it is possible to set interpacket delay to 10000 and the camera bandwidth to 60%, so the bandwidth is automatically set to 109Mbps (~87%).

Regarding the second suggestion, I divided the cameras into 3 groups and increased the transmission delay sequentially. but the dropped frames persists. Please take a look at the image below.

[kesaroid]

Another thing that I noticed which may be causing the issue is the Payload received and fps received vary drastically while running the tests and are not restricted to 55Mbps and 24.0 fps respectively. Do you know what is the reason for this, and if there is a way to restrict that?

[SMA2016a]

@kesaroid did you find any solution or you still need help to setup your system?

[kesaroid]

I found a way to limit the payload received by all cameras to 55Mb/s. This restricted the fps received as well. The Bandwidth is also optimized at 80Mb/s. I did this by setting the following parameters:

Jumbo packet: 1500 Inter-packet delay: 6912 and the Frame transmission delay is increments of 1528 for each subsequent camera.

All cameras are set to 24fps without packet drops (Errors).