javacv
javacv copied to clipboard
bytedeco
Major problem with Javacv when grabbing from camera & showing frames in JavaFX ImageView
Hi again, this time I've got a broader question, I don't know if this is JavaCV related but it feels like it might be and as I don't know where else to ask this I'm asking it here.
The thing is: I'm writing a JavaFX program that should show the frames from a camera and gives the option to also record the frames and the audio to a video. I'm on a Windows 10 Home laptop software version 21H2, if that matters.
I've had 2 times already when I was using an external camera (Logitech C615) and was recording at the time that my whole Windows crashed (see the picture for the look of my screen when that happened: 
). So I abandoned the external camera and thought "probably a bug in the driver for that camera", but just now the same happened with the webcam of my laptop when I wasn't recording a video, only grabbing & showing the frames in the front-end. In fact, the laptop screen wasn't even showing the JavaFX GUI, it was in the background as I was searching for a solution for some other issue I'm having in my browser.
This time I made a, very short, video. I hit the record button mistakenly twice right after each other:
https://user-images.githubusercontent.com/5166417/212095150-b0f792c7-b574-4e5d-96e4-907cda2f9dbf.MOV
Both after the second crash and after this third crash I've looked in Windows event viewer to see if something was logged there regarding "what went wrong just before the laptop restarted", but I could only find log events concerning the restart itself, not what happened in the minute before ...
As this setup of grabbing using JavaCV and showing the frames in JavaFX can be found online it seems others have got this combination working without big problems, but mine is having this showstopper :-(
Any idea what causes Windows to collapse in this way?
Make sure to use VideoInputFrameGrabber on Windows. If that's causing problems, try FFmpegFrameGrabber instead.
Thanx! Hopefully that will fix these problems. I was using OpenCVFrameGrabber until now
Actually, that doesn't work at all. Tried 3 recordings, both with the laptop internal camera and with an USB camera, all 3 make the Java application crash with something like this:
A fatal error has been detected by the Java Runtime Environment: EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffa2b4b5bb6, pid=20204, tid=16436 JRE version: OpenJDK Runtime Environment (19.0.1+11) (build 19.0.1+11) Java VM: OpenJDK 64-Bit Server VM (19.0.1+11, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) Problematic frame: C [ntdll.dll+0x25bb6]
No core dump will be written. Minidumps are not enabled by default on client versions of Windows
An error report file with more information is saved as: C:\dev\spike\hs_err_pid20204.log
If you would like to submit a bug report, please visit: https://bell-sw.com/support The crash happened outside the Java Virtual Machine in native code. Java VM: OpenJDK 64-Bit Server VM (19.0.1+11, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) Problematic frame: C [ntdll.dll+0x25bb6]
Would like to try FFmpegFrameGrabber, but that doesn't work with an index and also not with examples I tries like "Integrated", "desktop", "HD Webcam", "HD Webcam 0" or "video='HD Webcam'", as the name for the integrated webcam is HD Webcam (sometimes HD Webcam 0) The only other option I find when I Google which isn't a file or URL is “/dev/video0”, but that's the Linux way of device-description, kinda weird that the Windows device name doesn't also work then :-/
Something like this should work:
FFmpegFrameGrabber grabber = new FFmpegFrameGrabber("Integrated Camera");
grabber.setFormat("dshow");
grabber.start();
See https://trac.ffmpeg.org/wiki/DirectShow
Actually both frameGrabber = new FFmpegFrameGrabber("HD Webcam C615"); and frameGrabber = new FFmpegFrameGrabber("HD Webcam"); do not work in combination with frameGrabber.setFormat("dshow"); and frameGrabber.start();
It gives: Exception when trying to start grabbing from camera: avformat_open_input() error -5: Could not open input "HD Webcam C615". (Has setFormat() been called?) (For more details, make sure FFmpegLogCallback.set() has been called.)
It took some time, because there was something wrong with my environment. Apparently I was using code that was no longer supported in Java 19, a simple "mvn clean install" pointed that out but I had been running the software from Intellij for a while. I also had switched to Liberica JDK before and to know for sure that isn't the problem I installed OpenJDK 19 from sun, compiled and ran the jar from the command line and that gave me also the access violation:
EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff9841fb7c8, pid=17132, tid=15528
JRE version: OpenJDK Runtime Environment (19.0.1+10) (build 19.0.1+10-21) 13:32:42.305 [JavaFX Application Thread] INFO nl.politie.avroplaptop.CameraGrabber - Recording stopped Java VM: OpenJDK 64-Bit Server VM (19.0.1+10-21, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) Problematic frame: C 0x00007ff9841fb7c8
No core dump will be written. Minidumps are not enabled by default on client versions of Windows
An error report file with more information is saved as: If you would like to submit a bug report, please visit: https://bugreport.java.com/bugreport/crash.jsp The crash happened outside the Java Virtual Machine in native code. See problematic frame for where to report the bug.
Actually showing the video in JavaFX and recording it to disk works, this access violation happens when the recorder is stopped.
BTW: I do have ffmpeg on my system, but it's not on the Windows System Path. Is that maybe mandatory for the FFmpegFrameGrabber to work correctly?
Going back to the openJDK17 I used until a month ago gives the same EXCEPTION_ACCESS_VIOLATION (0xc0000005) with the FFmpegFrameGrabber.
With the VideoInputFrameGrabber together with openJDK 17 starting from the command line I don't get the access violation, but the application does close after the recording, which is also not as it should be. It should go to another screen after a recording is finished.
With both the VideoInputFrameGrabber and FFmpegFrameGrabber not working correctly on closing a recording I think the best that I can do is going back in the code to the OpenCVFrameGrabber in combination with openJDK 17 as I haven't had Windows crashes with that combination. Unfortunately the OpenCVFrameGrabber takes by far the most time to get a first frame on the screen from the three, but stability is more important than speed.
I must say I haven't done excessive testing with the OpenCVFrameGrabber / JDK17 combination, so I guess that starts now ...
Or you might still have a thought of a working setup with one of the other two Grabbers.
Either way, thanks for the fast responses and help so far :-)
It sounds like you're trying to use those objects from multiple threads. That's not a good idea. You should try to run everything from the same thread.
It was in fact even worse, I have code that gets called from a non-JavaFX-thread that executes a while loop with a while(someBoolean = true) statement and after that while loop grabber.stop and recorder.stop are called within the same function before it ends.
But ... the button that stops the recording not only sets the boolean that keeps that while-loop going to false, it also called grabber.stop and recorder.stop too, coming from the JavaFX thread.
facepalm
After removing that second call to both stop functions both FFmpegFrameGrabber and VideoInputFrameGrabber work, but ... frameGrabber.grab() with these two grabbers takes about 100 ms each time instead of about 30 ms for the OpenCVFrameGrabber, so 25 or 30 fps is not possible anymore. Is there some setting I need to set on the grabber that can speed this up?
The code with which I check the speed of the grabbing simply calls the external method:
long startTime = System.currentTimeMillis();
Frame frame = frameGrabber.grab();
System.out.println("Duration: " + (System.currentTimeMillis() - startTime));
It's probably just some setting that OpenCV sets by default but that the others don't...
Well I don't know, looking at the grab() function in the three Grabbers they're all quite different.
In VideoInputFrameGrabber it's the call myVideoInput.getPixels that takes around 100 ms per frame both when using the internal camera of the laptop or the USB-camera (and downloading the source for the videoInput.java file doesn't help me to debug further).
FFmpegFrameGrabber actually works in 30 ms per frame grab when using the internal camera of the laptop, but it's 50-100 ms per frame grab when using a USB-camera and ... it gives errors every call (I have the line 'FFmpegLogCallback.set();' in my code): Error: [mjpeg @ 000002a6f82aea00] unable to decode APP fields: Invalid data found when processing input Warning: [swscaler @ 000002a6c35b7240] deprecated pixel format used, make sure you did set range correctly
So, as the purpose of this app is grabbing video from a USB-camera under Windows I think I shouldn't go for FFmpegFrameGrabber. As you mentioned VideoInputFrameGrabber is preferred on Windows I'll use that.
With VideoInputFrameGrabber I tried a couple of settings that didn't seem to change anything for the duration of a framGrabber.grab() call:
//frameGrabber.setVideoOption("threads", "16"); //1? 4? 16?
//frameGrabber.setVideoOption("tiles", "16"); //4? 16? 128?
//frameGrabber.setVideoOption("tile-columns", "4");
//frameGrabber.setVideoOption("row-mt", "1");
//frameGrabber.setVideoOption("preset", "ultrafast");
//frameGrabber.setVideoOption("slice", "tiles");
and, actually, most settings I can find seem to influence the recorder, not the grabber, and it's the grabber that's too slow.
So I reverted back to my solution of a couple of weeks ago with a separate thread for every grabber.grab() surrounded by a Thread.sleep(40) (for 25 fps) just like I showed here: https://stackoverflow.com/questions/74512869/bytedeco-javcv-opencvframegrabber-grab-is-too-slow-for-25-fps
At least this works to get 25 fps. But, yeah, I guess I'll have to do extensive testing to see if this won't cause problems because there are different threads used when talking to the recorder.
Right, to prevent frames from getting dropped, we need to make sure the next call to grab() happens as quickly as possible after the previous call. So yes that means you should have a dedicated thread for the grabber, and not do any processing or call record() in there. OpenCV might be doing that internally, but VideoInput doesn't.
Well I must say my 'solution' didn't work either, it had less frames than it should have at 25 * 'the amount of seconds recorded' which meant the video (that did have 25 fps) had no frames left to show in the last seconds, which makes sense as a Thread.sleep(40) needs some time to execute as well and over time the lag builds up.
When I do all the grab() calls from the same thread (again) without sleep or other interference I'm stuck with about 100 ms per frame for the VideoInputFrameGrabber, so switching back to the OpenCVFrameGrabber seems logical. Unfortunately only the internal camera works fine with that, the first frame comes in after about 5 seconds (a little slower than VideoInputFrameGrabber, hat takes 1 second), but this is still 'okay' for an application and then the amount of ms that the grab calls take seems good most of the time as well: 32 - 31 - 32 - 31 - 31 - 32 - 62 - 32 - 31 - 34 - 30, etc.
But, with the USB camera it takes more than a minute for the first frame :-( And ...the amount of time between the grab() calls is now varying a lot: 2 - 77 - 64 - 3 - 3 - 58 - 4 - 60 - 2 - 60, although on average it's about the same, so that might still work out fine.
Anyway, the waiting time of more than a minute at the beginning is not acceptable. I wonder if this is some driver-issue, as the VideoInputFrameGrabber can get a first frame in about a second.
I'll try to see if I can download any driver for this camera, haven't installed one so far as Windows simply did configuration (and maybe downloading something, maybe not, I don't remember) itself when I first connected it
First I couldn't find a driver, but I did find the Logitech Camera Settings app. Starting that I saw one thing that helps: the default setting for the camera is "NTSC 60 Hz" and when I change that to "PAL 50Hz" the amount of time between the grab() calls becomes the same as for the internal camera: 31 - 47 - 31 - 32 - 32 - 30 - 31 - 31 - 33 - 32
Searching a bit more I did find a driver (actually for Windows 8, but it still works in Windows 10) and ... good news! Getting the first frame still takes over a minute with OpenCVFrameGrabber, but when I use the VideoInputFrameGrabber now both have fast first grab and faster grabbing :-)
So it was a driver issue that made the VideoInputFrameGrabber slower than the OpenCVFrameGrabber after all
Of course, again, I will do some extensive testing, but finally seeing the recommended grabber work fine on 30fps is a relieve
Damn, the old problem again, which I thought I had fixed by getting rid of the double stop-calls for the grabber and recorder. https://user-images.githubusercontent.com/5166417/213450138-bdb681de-06ab-4bec-8f68-1173369ce4a2.MOV
I am using a different driver for the camera now and the VideoInputFrameGrabber instead of the OpenCVFrameGrabber, but still Windows crashes when recording :-(
I do see that the starting of the recorder is still called from the Java FX thread, where the grabbing and stopping are happening in another thread. But it feels weird to me that that could cause this. If call 1 comes from the Java FX thread and call 2 and up all come from a dedicated thread why would it crash around the 20.000th call to the grabber in the same thread? Makes no sense ...
As this behavior started on January 3rd and I made a change in the code earlier that day to add some feature (and I had been using OpenCVFrameGrabber for weeks then already) I reverted that code change now. Again: I'll do extensive testing, but either way, if this fixes the Windows crashes or not I'll let it know.
It wasn't the code change from January 3rd. After that I thought it might be Intellij with some weird caching problem, so I tried starting from the command line, that wasn't the solution either Then I thought: maybe I should use a target jar from the command line that's built with "mvn clean install -Djavacpp.platform=windows-x86_64" instead of the simple "mvn clean install" I use most of the time and ... that seemed to work as I got a first recording done that lasted over an hour without problems, but with the second recording it still crashed my system after about half an hour.
Right now I feel like I should return to a code base from around the time I started using the javacv library and do (extensive) testing with that as I can see from all the recorded videos that are still on my system that I haven't really done much recordings longer than 10 minutes before the 3rd of January, so it might be that the code has been unstable for way longer. In that case I simply didn't notice it yet as I usually do tests for a couple of seconds or minutes and that has been working fine all the time, where the crashes seem to only be happening after at least 10 minutes of grabbing the frames.
Oh well, another day goes by without solution. Sometimes the life of a programmer really is trial and error ...
Like I said above, we shouldn't be using FrameGrabber or FrameRecorder from multiple threads. It doesn't matter if you get the synchronization right and everything, it's just not something that the OS and most codecs support well. Try to use them all from the same thread, that is one thread for a FrameGrabber, and another thread for a FrameRecorder.
Really? Okay, I'll try to figure that out.
With the reverting to old code the crashing has disappeared, which doesn't mean the produced videos work well, so I don't have a working solution for the whole set-up yet, but what I've basically got as code now (again) is the code from https://github.com/rladstaetter/javacv-webcam/blob/master/src/main/java/net/ladstatt/javacv/fx/WebcamFXController.java with right after the setVideoView(frame); a call to recorder.record(frame);
Well, after some sidesteps of doing unrelated things I just rewrote the code. Now I have one class in which all the calls to the grabber are, with inside a thread all the frame grabs. And another class with all the recorder calls and inside it a thread in which the recorder records the frames, but ... after a couple of minutes the screen / Windows still crashed.
In fact, when typing this a few minutes ago after a restart of Windows which I had forced by hand the screen went in limbo state again when I hadn't even started up Intellij / the Java program from the cmd line.
The external camera was connected to the USB-port, so this screen going limbo / the Windows crashes are not connected to Java or the code you or I have written, but they have to do with the external camera / it's drivers / how Windows deals with it.
I will keep the separate threads for the grabber and recorder. For now I will test further using the built-in webcam and won't connect this external camera anymore. The good news is: we hadn't decided yet which external camera we want to use with the software I'm writing, but we do know for sure this one won't be it ...
You can close this issue as it is not related to your library (or even Java).
Thanks for all the help along the way figuring this out.
I do have another remark (question?), which is this: I'm about to leave this approach as it doesn't seem to give me a stable fps for the grabbing:
new Thread(() -> {
setCameraActive(true);
AtomicInteger counter = new AtomicInteger();
long startTime = System.currentTimeMillis();
while (getCameraActive()) {
try {
// long timeBeforeGrab = System.currentTimeMillis();
// frameGrabber.setTimestamp(timeBeforeGrab);
Frame frame = frameGrabber.grab();
if (counter.getAndIncrement() % 150 == 0) {
LOGGER.debug("Timestamp {}th frame: {}", counter.get(), System.currentTimeMillis() - startTime);
}
Platform.runLater(() -> {
setVideoView(frame);
});
if (isRecordingStarted) {
mediaRecorder.processFrame(frame);
}
} catch (FrameGrabber.Exception fe) {
LOGGER.error("Exception when grabbing frame from camera: {}", fe.getMessage());
}
}
try {
frameGrabber.stop();
} catch (FrameGrabber.Exception fe) {
LOGGER.error("Exception when releasing frame grabber: {}", fe.getMessage());
}
}).start();
Here's why: the grabbing of the frames doesn't happen exact enough. The recorded video will always show 30.00 fps in Windows, but I can see by the log lines every 150th frame that the milliseconds part goes up over time and eventually the difference goes into seconds. When I take the time between the first and the last log line and divide it by the amount of frames, the actual recording had an fps of somewhere in between 29,97 and 29,99.
That's very close to 30, but not 30.
The weird thing is that the video where audio and video are in sync after more than an hour is the one where the fps was 29,97 according to the calculation of the log lines, so the one that was the most off ...
But as that is some sort of standard as well (I googled it) I tried to tell both the grabber and the recorder to work at 29,97 fps, but that didn't work. The recording made then also had a grabbing that took place with an fps of 29,98 and the audio off-sync.
I suspect that the fluctuation between 29,97 and 29,99 is because of the always fluctuating power grid frequency, which is usually just below 50 Hz (https://www.swissgrid.ch/en/home/operation/grid-data/current-data.html#wide-area-monitoring).
Otherwise I can't tell why the grabbing frequency fluctuates between recordings.
So, I'm thinking of using a ScheduledThreadPoolExecutor that checks for a new frame in the grabber 25 times a second as that should finally give me a steady grabbing of exactly 25 frames per second with log lines every 150 frames without the milliseconds part going up over time ...
I do wonder if, when I do that, the audio part will be on or off-sync though as the code that grabs the audio is already inside a ScheduledThreadPoolExecutor ...
You'll have a hard time to get exactly 30 FPS with a consumer grade device. You'll probably need to go professional for that like these ones: https://www.flir.com/browse/industrial/machine-vision-cameras
Well an exact fps isn't a requirement, but audio and video in sync even if the recording takes a couple of hours is, so I hope I can figure that out with a consumer grade device :-)
That we can easily do by dropping or duplicating frames from time to time when the lag goes over the length of an average frame.
I'm ditching a video frame now and then but somehow everything still messes up in the end.
I have one thread that in a while loop constantly asks the FrameGrabber to get the new frame and when it gets it stores it in a local variable I have another thread with a while loop an in it a Thread.sleep of 10 ms and every time it comes around checks the current time compared to the 'recorded frame counter' and the start time (plus counter times 40 ms) to see if it should send the current frame to the recorder (and counter +1) or not.
This way I should get exactly 25 frames per second. Several times a second ditching a frame that was taken from the grabber, and also sometimes using the same frame twice as the camera isn't always fast enough (i.e. less than 40 ms) with delivering a frame to the grabber. Tested it with debug-logging on a short recording and saw pass-pass-pass-record-pass-pass-pass-record for a while so that looked good.
But ... on my last try the recording I made took 09:49:11 (end time minus start time), but the recorded video is 'only' 09:48:59 long. So it misses about 12 seconds, or 300 frames on a recording. It should have been between 883775 and 883800 frames, but is between 833475 and 883500 :-/
I can actually see in the debug logging that around 934400 frames have been grabbed from the camera during the 9:49:11 of the recording, so there were more than enough frames to use.
How is this going wrong?
The only reason I can think of right now is that the recorder doesn't always store the frame I send to it, that it sometimes simply thinks "Already got this one, gonna ignore it", can this be the case?
Are you sure the codec you're using is fast enough to encode this in real time?
I didn't look into that, I simply used audio codec AAC, video codec H264 and file format MP4 and thought it would work. Where do you think the bottleneck is?
Well, it's actually a hardware issue. This device (a NexiGo N930AF) had this behavior on long recordings. This is a 50 Dollar device. I've also tested with a Dell WB5023 (a 100 Dollar device) which didn't have this missing frames issue on long recordings, but the audio and video where not completely in sync anymore after several hours. Then I tested with the Logitech Brio (a 150 Dollar device) and on this long recording the audio and video where still in sync after 7 and a half hours.
This doesn't mean the Brio is the definitive choice for us to use 'in production', because I want to test more and especially in a production-like-environment, but for now it's the best candidate.
So, yeah, it depends on the hardware how well the software can keep delivering frames and audio on longer recordings.