tiscamera
tiscamera copied to clipboard
TheImagingSource
Multiple cameras, iminent dropouts
I've been working with 2 DFK33UX264's for a couple years now constantly chasing down an issue with 1 or both cameras dropping out, usually very early after starting the capture. Camera is still seen as a USB device but images stop coming in. Releasing and reopening the cap usually gets it back but that's not a good solution. Cameras are controlled by external trigger, off an encoder and some logic but that's not really important, just to say that the framerate is never completely consistent but always lower than the max 15FPS @ 2448x2048. Cap is accessed by OpenCV and cameras are set through V4L2. Sometimes it seems like it's better, then it gets bad again. Ive been through multiple different small PCs running various Linux distros for other reasons related to the machine that this is on. Kept trying to iron out program details that may have been messing with the cameras, all sorts of things, powered USB3 hubs, etc.
I finally tried something different which I would have tried sooner but the PCs I use tend not to have full size PCIE slots. I tried a USB3 PCIE card with multiple Renasis controllers and each camera on it's own controller and the problem is gone. I've monitored these cameras with usbtop in the past and both of them running together doesn't consume anywhere near the full USB3 bandwidth limit, even on 10gb/s ports. The difference now is that I'm not using ports on the USB3 controller that's built into an AMD or Intel CPU (both are just as bad).
This may not be a problem specific to TIS cameras, maybe it's a Linux thing. I just find it odd that I never see this mentioned but I've seen this issue on enough different hardware to know it's not really specific to anything. I've seen this same thing reported on Nvidia Jetson's and many small form factor devices that typically won't have external USB3 chips and just use what comes off the CPU, or has a single chip on the board. Is it really necessary to go through such extremes as to put every camera on it's own controller if it's not a bandwidth issue?
Hello
First of all, many thanks for investigating into this issue and making so many tests,
In my personal experience, the Renesas chipset does not reach the full USB 3.0 speed, its speed is around 4000mBit/s. If the cameras are configured to run with 2448@2048 resolution at 15fps, then they will reach together 1430mBits/s for the 1 byte image formats and 2900 mBit/s for the 2 byte video formats. These values are estimated, but clearly within the bandwidth of the Renesas chip. Therefore, I guess, the 15 fps are the trigger frequency, while the cameras run a higher fps, e.g. 38 fps. This reaches the 75% USB 3 bandwidth for the one byte formats and much more for the two byte pixel formats. That leads to incomplete image data transfer and incomplete frames are dropped.
Therefore, I suggest to set the frame rate too: cap.set(CV_CAP_PROP_FPS, 17.0); 17 fps in order to be somewhat faster, than your trigger frequency. Then you can use longer exposure times. With that, one USB 3.0 controller, even a Renesas, should do the job.
However, that is guessing about frame rates, because I do not know, whether you set the frame rate or not.
The Intel CPU's USB 3.0 controller should work fine. We use them, as well as external USB 3.0 boards for firmware tests. The external USB 3.0 board employ a Fresco Logic chipset.
The AMD USB 3.0 controller could be critical, I have many issues with it on my Windows computer. Therefore, I do not think, the issue is a pure Linux thing.
NVidia Jetson should be fine. I made also some tests on a PI4 using v4l2 directly and figured out, that the USB 3.0 performance is quite well. ... as long as you do not do image processing in parallel. Then you get frame drops.
I will appreciate, if you let me know your thoughts.
Best regards
Stefan
Hi Stefan,
Let me back track a little. Until now, I've never used a Renasis chipset, all the problems I had were the USB3 controllers built into intel N and J series CPUs and for a short period I used the controller built into an AMD A10 APU, basically whatever the ports on the board were connected to. The Renasis chip is what solved the problem, not because it's the Renasis chip specifically but because you can buy cards with multiple Renasis chips on board.
I have no actual framerate in practice, the machine I use it on has an encoder that measures the position of something that rotates then an FPGA card triggers the camera when the position is reached. 15FPS is set because in triggered mode the FPS setting becomes the limit, at say 5FPS, the camera will only allow less frequent triggers. The point was that when triggering by machine my instance does not even actually hit "15FPS" because it doesn't need to. This problem occurs with the cameras in free running mode just the same so that is not really the point. I expose @ <300us because I take pictures of something that moves fast, My strobe is suitable, I don't need any longer exposure times. The highest framerate available @ @.*** is 15fps, I don't actually need it any faster than that but that's the camera's limit nonetheless. For testing I just leave the cameras in free-running mode @15fps and at least 1 will drop out without fail and never send another frame until it is restarted. With the current PC I use (a J4125) it happens very quickly when plugged into native USB ports, powered hub or not. With the card with multiple controllers I can leave it run all night and the cameras will still be running hours later on the same PC
I've seen the exact same thing reported on the Nvidia forums with Jetson Nano's and various other forums with random types of CPUs, nobody ever seems to figure out the problem. Again, it only occurs with multiple cameras connected, I've never really tested with a single camera because I require 2.
On Thu, Dec 30, 2021 at 2:57 AM TIS-Stefan @.***> wrote:
Hello
First of all, many thanks for investigating into this issue and making so many tests,
In my personal experience, the Renesas chipset does not reach the full USB 3.0 speed, its speed is around 4000mBit/s. If the cameras are configured to run with @.*** resolution at 15fps, then they will reach together 1430mBits/s for the 1 byte image formats and 2900 mBit/s for the 2 byte video formats. These values are estimated, but clearly within the bandwidth of the Renesas chip. Therefore, I guess, the 15 fps are the trigger frequency, while the cameras run a higher fps, e.g. 38 fps. This reaches the 75% USB 3 bandwidth for the one byte formats and much more for the two byte pixel formats. That leads to incomplete image data transfer and incomplete frames are dropped.
Therefore, I suggest to set the frame rate too: cap.set(CV_CAP_PROP_FPS, 17.0); 17 fps in order to be somewhat faster, than your trigger frequency. Then you can use longer exposure times. With that, one USB 3.0 controller, even a Renesas, should do the job.
However, that is guessing about frame rates, because I do not know, whether you set the frame rate or not.
The Intel CPU's USB 3.0 controller should work fine. We use them, as well as external USB 3.0 boards for firmware tests. The external USB 3.0 board employ a Fresco Logic chipset.
The AMD USB 3.0 controller could be critical, I have many issues with it on my Windows computer. Therefore, I do not think, the issue is a pure Linux thing.
NVidia Jetson should be fine. I made also some tests on a PI4 using v4l2 directly and figured out, that the USB 3.0 performance is quite well. ... as long as you do not do image processing in parallel. Then you get frame drops.
Best regards
Stefan
— Reply to this email directly, view it on GitHub https://github.com/TheImagingSource/tiscamera/issues/443#issuecomment-1002913690, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZQMOUGP4QWPKFNGBKVRSDUTQGHNANCNFSM5K62DNWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID: @.***>
Hello
The frame rate in the camera determines, how fast the image data packets are sent out. Therefore, it is important, even if you run with a lower trigger frequency. The lower the frame rate, the longer the pauses between the image data packtes. Thus, the risk of a buffer overflow and packet lost in the camera is less.
I must admit, I did not test multiple USB cameras on NVidia hardware, at least, I do not remember doing that.
Just to confirm: Did you set the 15 or the next higher frame rate? If not, the camera will use maximum frame rate and this wont work, if two of them are connected to one USB controller.
Stefan
I must admit, I did not test multiple USB cameras on NVidia hardware, at least, I do not remember doing that.
We're only talking about the same thing if you're using more than 1 camera
Just to confirm: Did you set the 15 or the next higher frame rate? If not, the camera will use maximum frame rate and this wont work, if two of them are connected to one USB controller.
I don't understand what you are saying here, it sounds like you're suggesting this is a bandwidth thing. Let's forget about the "triggered" mode, that has no effect as most testing is done in free-run. The cameras....both are set for BGR3 2448x2048 at 15fps, that never changes. That along with all parameters are set to the camera everytime I start the program. usbtop reports <250,000kb/s on each camera with the above settings free-running. You said above that this format should produce ~1400mb/s which I don't understand, either usbtop is completely inaccurate or something.
The below screenshot, taken in program with both cameras running @ 15fps shows the usbtop report. This is with the multi-controller card as that's how I have it now but this is pretty consistent with what I've always seen, except both cameras are on separate bus's now. Bus4/device2 and Bus6/device2 are the cameras.
parameter readouts: TIScam_v4l2.txt
Hello
With setting 15 fps I mean open CV code :
cap.set(CV_CAP_PROP_FPS, 15.0);
This configures the camera to send the images slower. Doing so, you should be able to run two cameras in parallel connected to only one USB controller. That is a bandwidth thing, you are correct here. But ... BGR3 works against it again, because there are three bytes per pixel transferred. Using BGR3 is a new information for me. Thus, the maximum FPS is indeed 15 instead of 38 fps. And therefore, you can not use one USB controller for both cameras.
You can reduce the bandwidth by using GRAY8 with 15 fps and use cvtColor for debayering. That will cost CPU load, but you may get better results on PI4 and NVidia hardware with both cameras.
Stefan
I know it's been awhile but this is indeed a bandwidth issue on a USB3.1 Gen 1 port. It seemed to allude me since for one usbtop has been lying to me and for two I suppose I expected a bandwidth issue to degrade performance, not completely stall the capture.
After using it for a while the problem does indeed go away with a card that can keep each camera on it's own controller but I don't necessarily like the form factor of having to use a PCIe card.
Have you ever tested or do you suspect 2 cameras would run fine on a single 10gbps USB3 controller? I haven't seen it explained anywhere whether multiple 5gbps devices benefit from the extra bandwidth of a 10gbps controller. I would suspect they do but have you tested this?
Hello
Have you ever tested or do you suspect 2 cameras would run fine on a single 10gbps USB3 controller?
I do not think, that will work, but I am not sure. I expect the 10GB USB controller switch down to 5GB again. However I really do not know.
Yes im sure it will switch to 5gbps for a single device, but the bandwidth on the controller should allow 10gbos regardless. A USB 3.0 5gbps controller requires 1 pcie2.0 lane. A 10gbps controller requires 2 pcie2.0 or 1 pcie3.0 lane depending on the chip itself. So the extra bandwidth is available to the controller with 10gbps chips.
If this something you've never tested, what is it that you guys recommend customers to do? Bandwidth limitations were never mentioned to me when I picked up my cameras, actually took quite a while to realize it was a problem at all.
On Mon, May 16, 2022, 6:42 AM TIS-Stefan @.***> wrote:
Hello
Have you ever tested or do you suspect 2 cameras would run fine on a single 10gbps USB3 controller?
I do not think, that will work, but I am not sure. I expect the 10GB USB controller switch down to 5GB again. However I really do not know.
— Reply to this email directly, view it on GitHub https://github.com/TheImagingSource/tiscamera/issues/443#issuecomment-1127510504, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZQMOROLWK3YY3V3UWRDS3VKIRBBANCNFSM5K62DNWA . You are receiving this because you authored the thread.Message ID: @.***>
We recommend to use either one USB controller per camera or reduce the frame rate.
I am sorry, that nobody mentioned bandwidth issues. But you could have asked earlier and direct your camera reseller.
Stefan
Do you consider "The Imaging Source" in America a "Reseller"? The guys in the US have never been very knowledgeable about the technical details of your products. There used to be a "The Imaging Source" forum to figure these things out, no idea what happened to that.
Not sure what I would have asked when I didn't realize it was a bandwidth issue. If the recommendation is to use multiple controllers you should post that somewhere clearly. A degradation of performance would make bandwidth issues obvious. Complete dropouts will have you chasing your tail.
The colleague should have knowledge.
The forum was closed more than 10 years ago. There camera too much spam in.
You can contact us directly from https://www.theimagingsource.com/company/contact/, then the conversation will run by email.
Stefan
