Problem with Buffer Mismatch

[xulingtian]

Describe the bug Hi, in our project I built the aravis Interface. Everything is running well except we have ARV_BUFFER_STATUS_SIZE_MISMATCH Situation with every camera. Some images are just missing a few pixels, but some of them are half black.

To Reproduce ArvBuffer *buffer = arv_stream_timeout_pop_buffer(m_stream, 1000000); some of return value of arv_buffer_get_status(buffer) is ARV_BUFFER_STATUS_SIZE_MISMATCH.

and m_stream is from m_stream = arv_camera_create_stream (m_camera, NULL, NULL, &m_error);

Expected behavior I hope every frame should have no Buffer Mismatch Problem.

Camera description:

• We use 4 camera together: 3x Basler daA1600-60uc (or daA3840-45uc) + 1x Basler daA1440-220um
• 2 cameras use one bus, another 2 cameras use another bus
• Interface: USB3

Platform description:

• Aravis version: 0.8.22
• OS: Ubuntu 16.04
• Hardware: x86_64
• cpu: Intel(R) Core(TM) i3-9100 CPU @ 3.60GHz

Additional context I have also get the output from m_stream using arv_stream_get_statistics(m_stream, &n_completed_buffers, &n_failures, &n_underruns); in our project: n_completed_buffers 324, n_failures 4, n_underruns 13

I just wonder if there are some parameter in Aravis should I adjust to improve the behaviour.

This is first time I use github Issues and maybe the description is not good enough. If there is anything you don't understand well, please let me know.

[xulingtian]

@EmmanuelP Can you please take a look and give me some advices? I will provide any Information you need. Thank you very much!

[langenf1]

Experiencing the same problems using an Alvium 1800 U-500C. Did you find a solution yet?

[EmmanuelP]

The number of underruns indicates the stream receiving thread runs out of available buffers to store the incoming data. You may try to increase the number of buffer you put in the queues. It could also indicates you spend to much time processing the images.

[EmmanuelP]

You may also want to run your code with ARV_DEBUG environment variable set to all and report the output here.

[xulingtian]

Hi @EmmanuelP , thank you for your respond!

I've tried the idea of increasing the number of buffers, but it doesn't work. And we have more Problems with the Basler daA3840-45uc compared with daA1600-60uc.

we have done two tests and get the outputs with ARV_DEBUG=all:

1. 3x daA3840-45uc + 1x daA1440-220um ( direct in Project ) daA3840_program_debug.txt
2. 1x daA3840 (a minimal standalone program) daA3840_minimal_standalone_debug.txt

The both has the Problem. And sometimes there is also an error when I want to set ROI of daA3840 [OffsetX] [d_898] [d_464] USB3Vision write_memory error (write-protect)

Please take a look at the files and I hope they are what you want. Thank you again.

[EmmanuelP]

From now on, it will be difficult to help without having a device at hand.

The TIMEOUT_ERROR messages are worrying. Do you have other devices connected to the USB bus ? Did you try to enable the asynchronous USB mode ?

https://aravisproject.github.io/aravis/usb.html#performance

What is the output of arv-camera-test-0.8 --debug=all --duration=5 ?

[xulingtian]

Hi,

• there are no other devices connected to the USB bus. Only the cameras are connected.

• I have tried the asynchronous USB mode.

  • For the standalone program, the Timeout Error disappears. But the payload is still always smaller than expected.daA3840_async_usb.txt
  • But for our Project with 4 cameras, the program always stops with the [OffsetX] [d_898] [d_464] USB3Vision write_memory error (write-protect) . Do you know what does it mean? The setROI Method should be right and OffsetX is also within the range. Sometimes the program works after usbreset or reboot, sometimes not.

• output of arv-camera-test-0.8 --debug=all --duration=5: daA3840_arv-camera-test-0.8.txt And this is output of ./arv-camera-test-0.8 --debug=all --duration=5 --usb-mode=async using async usb mode daA3840_arv-camera-test-0.8_async.txt

• If you want and it is allowed, we are happy to send you the camera you want. Just tell us how to send you. :)

[xulingtian]

Sorry, yesterday I clicked wrong and closed it. The issue is reopened

[xulingtian]

@EmmanuelP I would be very grateful if you have other solutions or other suggestions. As said above we can provide you with cameras if needed. Thank you very much.

[langenf1]

We had a similar issue and the crux of our problem seemed to be we were accidentally using an USB 2.0 port. Check your maximum bandwidth before starting your stream to check if you're limited by your bandwidth. Ours was 50MB/sec on 2.0 which came down to around ~21 FPS with our resolution and also caused many empty buffers with the mismatched buffer size status to get in our buffer pool (I presume because we set the camera FPS to 60 but we were limited to only 21 by bandwidth). On a 3.0 port we could significantly increase this bandwidth to match our desired framerate of 60 FPS (width x height x fps bytes/sec). This solved most of our issues. It also helps to check for arv_buffer_get_status(buffer) == ARV_BUFFER_STATUS_SUCCESS to only get the complete frames.

[xulingtian]

@langenf1 Thank you for your suggestion. We have considered the Possibility before. The USB Ports are 3.0. And we did a lot of tests by changing the bandwidth and DeviceLinkThroughputLimit but it doesn't work. arv_buffer_get_status(buffer) returns always lots of Mismatch Message. May i ask what kind of camera are you using?

[langenf1]

@xulingtian We are using the Allied Vision Alvium 1800 U-500C. Are both the port and the cable USB 3.0? Because a 2.0 cable would give the same issues. What is your bandwidth and desired resolution/framerate?

[EmmanuelP]

@EmmanuelP I would be very grateful if you have other solutions or other suggestions. As said above we can provide you with cameras if needed.

Hi @xulingtian

That would be nice. Please contact me privately at emmanuel.pacaud at free.fr .

[xulingtian]

@langenf1 Yes, the port and cable are both USB3.0. We have used pylon SDK from Basler before for several years. The Cable and Port are the same but we didn't meet such problems.

• The bandwidth we set currently is 50000000
• the FPS we set is 25, that is what we want
• We set the ROI of two cameras with 2400x1800 and set ROI of one camera with 3610x2178. Then we resize all of them to 1600x1200.

What's more, I have also a Allied Vision Alvium 1800 U-319c in hand and I tried it just now. Despite we can get the Frames with our code (most of them are n_f), but with arv-camera-test-0.8 it doesn't work at all and gives the error back:

Failed to read the current device configuration: [BinningHorizontal] [RegBinningHorizontal] USB3_Vision read_memory error (access-denied)

lsusb:

Bus 004 Device 021: ID 1ab2:0001 Allied Vision Vision Device

lsusb -t:

/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 20000M/x2 |__ Port 2: Dev 21, If 0, Class=Miscellaneous Device, Driver=, 5000M |__ Port 2: Dev 21, If 1, Class=Miscellaneous Device, Driver=, 5000M`

[xulingtian]

@EmmanuelP I would be very grateful if you have other solutions or other suggestions. As said above we can provide you with cameras if needed.

Hi @xulingtian

That would be nice. Please contact me privately at emmanuel.pacaud at free.fr .

@EmmanuelP thank you very much.

[langenf1]

@xulingtian 50000000/2400/1800 means you will get a maximum of ~11 FPS. This is exactly the same issue we faced. Furthermore, the bandwidth u listed is exactly the same as we got on a USB 2.0 port (USB 3.2 yielded 450.000.000 instead of 50.000.000). So either you have a faulty USB 3.0 port or you’re somehow connecting your device in USB 2.0 mode if I had to guess. Please double check your code with another cable and/or device/USB port if you haven’t already.

[xulingtian]

@xulingtian 50000000/2400/1800 means you will get a maximum of ~11 FPS. This is exactly the same issue we faced. Furthermore, the bandwidth u listed is exactly the same as we got on a USB 2.0 port (USB 3.2 yielded 450.000.000 instead of 50.000.000). So either you have a faulty USB 3.0 port or you’re somehow connecting your device in USB 2.0 mode if I had to guess. Please double check your code with another cable and/or device/USB port if you haven’t already.

@langenf1 actually after testing with minimal standalone Code, I get 18 FPS with resolution 3840x2160 (daA3840, with resize 2400x1600, I get FPS 20).... But thank you for your advice, I will test them again

[langenf1]

@xulingtian I’m talking about complete frames, not mismatched ones included. If that’s your actual bandwidth then it’s not possible to get more than 11 complete frames per second at 2400x1800, 13 FPS at 2400x1600 or 6 FPS at 3840x2160 (complete frames only).

[xulingtian]

@langenf1 Ok i understand what you mean, thank you for your explain. I am just confused, with your conclusion the fps will never catch up to 25 whether the interface is Aravis or Pylon. But we didn't have the Problem when using Pylon.

We set the DeviceLinkThroughput of 250e6 in our project. Maybe it helps the improvement of FPS?

[langenf1]

@xulingtian Please use arv_camera_uv_get_bandwidth_bounds() and tell me what the output is. If the maximum is 50M then you’re connected through USB 2.0 in some way, I’m not sure how, that’s up to you to find out. Could you also make sure arv_camera_uv_is_bandwidth_control_available() is true?

[xulingtian]

@langenf1
bandwidth_bounds: min: 524288 max: 419430400 Return value of arv_camera_uv_is_bandwidth_control_available() is false.

[langenf1]

@xulingtian That’s strange, it appears your camera’s bandwidth can’t be set? Your maximum bandwidth is definitely sufficient and indicates you are indeed connected a 3.0+ port. What is the output of arv_camera_uv_get_bandwidth() and arv_camera_get_payload()? Do you get any errors on arv_camera_uv_set_bandwidth()?

And can you confirm if getting, setting (to some random value within the bounds) and then getting the bandwidth again actually changed the value or not?

[xulingtian]

@langenf1 The output of arv_camera_uv_get_bandwidth()in the beginning: 50000000

Then i use arv_camera_uv_set_bandwidth to set it to 80000000, the output of new arv_camera_uv_get_bandwidth() is 80000000. So i guess the funktion of setBandwidth works. And i get no error or warning of setting bandwidth.

output of arv_camera_get_payload() is 8294400 (3840x2160)

[langenf1]

@xulingtian Ur desired FPS is 25 right? Please set your camera frame rate to 25 and set your bandwidth to payload x 25 (8294400*25). With these two parameters set you should be good to go. Let me know what the results are and if you’re still getting mismatched buffers.

[xulingtian]

@langenf1 It looks good, the good news is that all the frames are ARV_BUFFER_STATUS_SUCCESS in the standalone Programm and I didn't see the mismatch Problem. But the fps is only 18.

But thank you very much for your help. I will try to test it again tomorrow in our large project. If I still meet the Problem, I will update it here.

[langenf1]

@xulingtian Good to hear 👍 Another factor that could limit your FPS could be that your exposure time is too high (if you're testing in a dark environment). Your maximum exposure time should be 1e6/fps so 40000 µs in your case.

[xulingtian]

@langenf1 however the large Project still has the Mismatch error after I set the Bandwidth. I think the differences are:

• in standalone project it runs well with only one Camera, but in large project I need 4x cameras working simultaneously (i don't know if it has effect)
• i test standalone project in my laptop. I need to run the large project in another device (not as good as a laptop).

I will discuss with my colleagues about the usb ports of the device again when they come back from vacation next week.

[langenf1]

@xulingtian what kind of device are you running on? Also, I haven't tested with multiple cameras at the same time so I can't really help you with that I'm afraid.

[xulingtian]

@langenf1 The device is Intel(R) Core(TM) i3-9100 CPU @ 3.60GHz. I will try to build the standalone Programm in the device and to see if it also has Mismatch Error. But I think the Possibility is very small that the device causes the problem, and I am sure that USB3 is used

Thanks anyway, you helped me a lot :)

[EmmanuelP]

Hi,

I have tested the latest aravis release (0.8.28) using 2 Baslers camera (daA1920-160um and daA3840-45u) using my laptop (DELL Latitude 5300 Intel® Core™ i5-8365U × 8), with 2 instances of arv-camera-test-0.8 running at the same time:

./src/arv-camera-test-0.8 -n Basler-267601CB75AB-40257834 -b 300000000
Looking for camera 'Basler-267601CB75AB-40257834'
vendor name            = Basler
model name             = daA1920-160um
device serial number   = 40257834
image width            = 1920
image height           = 1200
horizontal binning     = 1
vertical binning       = 1
exposure               = 5000 µs
gain                   = 0 dB
payload                = 2304000 bytes
163 frames/s -     376 MiB/s
...
164 frames/s -     378 MiB/s
164 frames/s -     378 MiB/s
^C165 frames/s -     380 MiB/s
n_completed_buffers    = 683118
n_failures             = 0
n_underruns            = 0
n_aborted              = 0
n_transferred_bytes    = 1575302898688
n_ignored_bytes        = 0

./src/arv-camera-test-0.8 -n Basler-267601CBA4F5-40281429 -b 300000000
Looking for camera 'Basler-267601CBA4F5-40281429'
vendor name            = Basler
model name             = daA3840-45uc
device serial number   = 40281429
image width            = 3840
image height           = 2160
exposure               = 15000 µs
gain                   = 0 dB
payload                = 8294400 bytes
 43 frames/s -     357 MiB/s
...
 42 frames/s -     348 MiB/s - 2 errors
 45 frames/s -     373 MiB/s
...
 45 frames/s -     373 MiB/s
 45 frames/s -     373 MiB/s
^C 45 frames/s -     373 MiB/s
n_completed_buffers    = 188633
n_failures             = 2
n_underruns            = 0
n_aborted              = 0
n_transferred_bytes    = 1565004663808
n_ignored_bytes        = 0

0 error on one camera, 2 errors over 188633 completed buffers on the other, with a constant 740MB/s USB transfer, during more than 1 hour. The CPU use was about 22-25% of one core for each arv-camera-test-0.8 instance.