WestRidgeSystems
Synchronization errors (lag) between video and metadata
Describe the bug In some mpeg.ts files I'm processing with jmisb I've noticed that the metadata stream seemed to be ahead (about 7.5 seconds (or so)) of the imagery.
I think it may something similar to the jmisb gitlab issue here;
https://github.com/WestRidgeSystems/jmisb/issues/419
I've also seeing the following exception once at the beginning of decoding some, but not all, of the files;
org.jmisb.api.video.MetadataDecodeThread - KLV parse exception
org.jmisb.api.common.KlvParseException: Invalid classifying country coding method: 0
at org.jmisb.api.common.ThrowOnInvalidDataStrategy.process(ThrowOnInvalidDataStrategy.java:17)
at org.jmisb.api.common.InvalidDataHandler.handleInvalidFieldEncoding(InvalidDataHandler.java:127)
at org.jmisb.api.klv.st0601.UasDatalinkMessage.
After that I get a large number of onMetadataReceived() callbacks before I get my first callback to onVideoReceived(). Depending on which file, I get 93, 31, 33, 37, etc metadata frames before I get the first video frame.
To Reproduce Unfortunately I'm unable to share the video files I'm using.
I've downloaded the 4 sample .ts files linked by YannPara, the author of the above issue, and also the 7 files linked to by the user bradh in the thread of that issue, and run them through a small test program I put together to see if they also had problems with synchronization/lagging video frames.
Of those 11 videos, only one seems to have a similar problem (although most of them logged a bunch of errors when processed, and some of them seemed to take a lot of time to open.)
klv_metadata_test_async.ts
It took 96ms for VideoFileInput.open() to complete.
1ms after open returned until first metadata frame.
52ms after open returned until the first video frame, by which time there had already been 98 metadata callbaks.
357 metadata frames in the first 200 video frames (the test program exits after the 200th video frame.)
This looks a lot like the sync problems I'm seeing with our files.
Also, there was this error;
[main] WARN org.jmisb.api.video.FfmpegLog - [mpegts @ 0x7f8e9f8b6400] start time for stream 0 is not set in estimate_timings_from_pts
And there were 99 of these errors;
[MetadataDecodeThread - klv_metadata_test_async.ts - 0] WARN org.jmisb.api.klv.st0102.ObjectCountryCodeString - Object Country Codes has too few bytes for required UTF-16 encoding. Trying UTF-8 workaround.
Expected behavior For videoframes and metadata to stay in sync.
Screenshots N/A
Configuration (please complete the following information): OS X 12.6
java version "1.8.0_291" Java(TM) SE Runtime Environment (build 1.8.0_291-b10) Java HotSpot(TM) 64-Bit Server VM (build 25.291-b10, mixed mode) jMISB 1.11.0
Additional context N/A
There are multiple issues here. I'll try to break it up with different responses, which might enable threading.
The part about
[MetadataDecodeThread - klv_metadata_test_async.ts - 0] WARN org.jmisb.api.klv.st0102.ObjectCountryCodeString - Object Country Codes has too few bytes for required UTF-16 encoding. Trying UTF-8 workaround.
is a warning (not an error).
The problem is that the MISB defined that as UTF-16, but most implementers got it wrong. They just dropped ASCII text in because that is what is used everywhere else in ST 0102. We have a heuristic if there are insufficient bytes for it to be UTF-16, but given we're in undefined behaviour land, I didn't want to just ignore it. You can obviously adjust your logging to filter that out.
The part about
org.jmisb.api.common.KlvParseException: Invalid classifying country coding method: 0
looks like bogus data. You can't use 0 (see ST 0102 Table 2). Doing so invalidates the classification country code.
If you can't fix your data, you can choose not to throw an exception. See https://westridgesystems.github.io/jmisb/org/jmisb/api/common/LogOnInvalidDataStrategy.html
The offsets are likely driven by not providing good PTS info. Are you providing precision time stamps in the SEI and metadata? Can you possibly align off that instead?
The jmisb implementation uses presentation time stamps (not precision time stamps), so if the PTS isn't valid, you may not get good alignment. What do you see in tsduck or equivalent tool? Can you show the headers out of tsduck (even if you can't provide a full dump)?
Please understand that we may not be able to address this without seeing the data.
I do understand the difficulty in not being able to see the data. I suspect I'm seeing similar behavior for the sample file klv_metadata_test_async.ts that you linked to in issue #419 so I was hoping that might help.
I'm running an extraction right now to get the videoFrame and metadataFrame presentation timestamps on all of my video files, just to be absolutely certain, but I'm pretty sure the value for metadataFrame PTS I'm seeing is always the value;
-102481911520608.62000000
I'm not sure if this is in the files or something going on with jMISB.
All of my videos have what seems to be appropriate VideoFrame pts values, but every single MetadataFrame.pts in every file is -102481911520608.62000000.
The same is true for at least the first 200 frames of the sample video file klv_metadata_test_async.ts.
I'm currently running a check on every frame of all of the sample files.
And I'm checking out tsduck now.
All of the sample linked from issue 419, except for klv_metadata_test_async.ts, have what appears to be good values for MetadaFrame.pts.
The sample klv_metadata_test_async.ts, linked to from issue 419, and all of my video files have a value of -102481911520608.62000000 for every single MetadataFrame.pts. I don't know if this is an error in the file or something jMISB is doing, would be nice to verify if it's the files. (I'm still looking at the tsduck utility, trying to see if it can tell me this.) If it's an error in the files then I may be able to go back to the folks that supplied them to me and ask if they can rectify the problem or supply me with video files that don't have this problem.
My files, along with klv_metadata_test_async.ts, all seem to have valid VideoFrame.pts values - at least they don't seem egregiously wrong to the eye. The metadata in my files does have the precisionTimeStamp field, can you point me to documentation (or perhaps good search keywords) on how to align frames using the metadata precisionTimeStamp and the VideoFrame.pts?
All of the sample linked from issue 419, except for klv_metadata_test_async.ts, have what appears to be good values for MetadaFrame.pts.
OK. Maybe your data is async encoded, hence no PTS. (See MISB 1402 and SMPTE RP 217).
The sample klv_metadata_test_async.ts, linked to from issue 419, and all of my video files have a value of -102481911520608.62000000 for every single MetadataFrame.pts. I don't know if this is an error in the file or something jMISB is doing, would be nice to verify if it's the files. (I'm still looking at the tsduck utility, trying to see if it can tell me this.) If it's an error in the files then I may be able to go back to the folks that supplied them to me and ask if they can rectify the problem or supply me with video files that don't have this problem.
See above for whether this is expected or not.
My files, along with klv_metadata_test_async.ts, all seem to have valid VideoFrame.pts values - at least they don't seem egregiously wrong to the eye. The metadata in my files does have the precisionTimeStamp field, can you point me to documentation (or perhaps good search keywords) on how to align frames using the metadata precisionTimeStamp and the VideoFrame.pts?
I don't this is going to work in general. ISO 13818 describes how presentation and display time stamps work, but that is not absolute. There should be a precisionTimeStamp in the video frame SEI, but we don't surface it in 1.11.0, and some data providers don't add it.
Can you say what the goal of your work is? What analysis are you trying to do? Maybe there is another way we can help you move forward.
We are using geolocation fields in the metadata to render graphics over the video stream. For instance, given we know about an entity at a location in the world via some other means, when the video stream field of view is on that entity we want to overlay the display of an icon at the correponding location.
I've tried setting the InvalidDataHandler to LogOnInvalidDataStrategy but this doesn't seem to affect the videoframe/metadataframe 'lag'.
I am so far unable to find a copy of ISO 13818 or SMPTE RP 271, but I've read over MISB ST 1402.2. I believe particularly section 9.4.2, "Asynchronous Metadata Multiplex Method" may be what you intended me to see?
If my videos are asynchronous, shouldn't the metadata frames still be interleaved with the video frames?
I'm seeing a number (typically 30 to 90) of metadata frames before I see the very first video frames. Are these metadata frames simply all that's there in the initial part of the file (or in the file klv_metadata_test_async.ts), or are the video frames being delayed somehow due to other errors?
(I'm still trying to figure out tsduck, the docs are thorough but quite large. I have managed to use tsp pcrextract on my files but I can share that output with you if it would be helpful. )
You can purchase ISO standards from ISO or IEC, or your local distributor (e.g. national standards body, depending on country). You can purchase SMTPE standards from SMPTE and probably other distributors. ITU H.222 is very similar to ISO 13818, and would probably give you insight into the timing approach.
I didn't write the implementation of that part of jmisb, but assume if you modify the code to work off precision time stamps that you could align them that way. I do have an implementation in the 2.x branch that will extract the time stamp from the video SEI if present. See https://github.com/WestRidgeSystems/jmisb/blob/2.x/api-ffmpeg/src/main/java/org/jmisb/api/video/VideoFrame.java#L53