obsproject
libobs, obs-x264: MISP precision timestamps
Description
Introduce MISP precision timestamps, based on the following open standards:
- https://www.itu.int/rec/T-REC-H.264
- https://gwg.nga.mil/misb/docs/standards/ST0603.5.pdf
- https://gwg.nga.mil/misb/docs/standards/ST0604.6.pdf
This PR attaches MISP precision timestamps to the x264 encoder to introduce this new module. There is a follow-up PR ready which will attach MISP timestamps to Jim's NVENC encoder. Based on this design, attaching MISP timestamps to any OBS encoder should only require a few lines of code.
Motivation and Context
This change allows downstream players to synchronise incoming OBS streams for which there are countless applications. One of those would be esports tournaments where multiple people observe matches and stream their observations into a production control room. The ability to synchronise these incoming streams means that switching between streams would not create unwanted "jumps in time".
How Has This Been Tested?
The implementation of the specification has been tested on Mac and Windows by streaming to an RTMP endpoint and reading from it using ffmpeg:
ffmpeg -i [...] -c:v copy -bsf:v trace_headers -f null -
- Mac: BigSur 11.1 on a MacBook Pro 8GB
- Windows: Windows 10 on an AWS'
g4dn.2xlarge
The exact tests that were run:
- Enable/disbale sending of timestamps before/during streaming
- Update UTC<>TAI difference before/during streaming
- Change system time before/during streaming (observe attaching the correct direction of discontinuty)
Types of changes
- New feature (non-breaking change which adds functionality)
Checklist:
- [x] My code has been run through clang-format.
- [x] I have read the contributing document.
- [x] My code is not on the master branch.
- [x] The code has been tested.
- [x] All commit messages are properly formatted and commits squashed where appropriate.
- [ ] I have included updates to all appropriate documentation.
I would be interested in your thoughts on whether the new options for the x264 encoder should be officially documented somewhere or whether they can just be put in there like that; either way is perfectly fine for me 😊
Per a discussion earlier this month in the Discord channel for reference:
As far as we know, there currently is no public, free media player that supports using these timestamps. This means that ffmpeg is currently the only way to actually test this PR.
The current hope is that the existence of this PR / this feature in OBS might incentivise someone to either write such a player, or to integrate MISP timestamp handling into the OBS Media Source for incoming streams.
It's worth noting that a completely alternative way of accomplishing something similar would be to embed timecodes in the RTMP stream instead of the h264 video stream (I believe using onFI cuepoints). Obviously this means you can only use the feature with RTMP, but the advantage is that there are at least some commercially-available products/services that already support using RTMP timestamps to sync streams, such as Wowza. I think both solutions to the problem are viable and have different advantages and disadvantages, and there's probably room to have both in OBS, but the more immediately-useful solution would use RTMP timestamps.
Outside of coding complexity, any downsides to having the MISP data in both the encoded video and streaming container?
why_not_both.jpg
Outside of coding complexity, any downsides to having the MISP data in both the encoded video and streaming container?
On an "offer" level I think it'd be great to offer a wide variety of different options on all levels of the stack as different downstream services will likely have different things they'll accept/recognise/prefer. On the usage side, you obviously want to be careful to not mix and mingle these things but make sure users will only (carefully) select the kind of timing information that is recognised/accepted by a downstream player.
For example, when it comes to onFI, my understanding is that these are time codes (potentially drop codes). I don't think there is any obligation to synchronise these time codes with any specific clock (happy to be corrected). If you have a downstream player that interprets these onFI messages as UTC but you're actually sending MISP time, and you're also providing MISP precision timestamps, downstream players will need to decide which of these timing sources to use.
This more general issue with time codes (not necessarily being synchronised to a specific clock per specification) also lead us to preferring timestamps as you will be able to use these across different incoming streams/providers. In the grander scheme of things, you can take timestamps at face value (trusting that they are NTP/GPS/... synchronised) and understand their absolute (not relative!) offset universally.
As for RTMP, I believe there are also some AMF message types with timestamps (not time codes) but they are mostly vendor specific (again, happy to be corrected).
@cherusker i'm unearthing this PR. What would be your intents with it ? Imo the purported goal is interesting although in its current state it's only one side of the coin (encoding the SEI timestamps). What would tremendously increase the interest in the PR is if syncing of obs Media Sources was implemented, because it's a feature we've very much wanted in obs for a while.
I think this is a great addition. Twitch would definitely use these broadcaster wall clock timestamps.
We currently we insert our own SEI timestamp the moment the frame is received on our network. It's a dramatic improvement to report the timestamp before the frame was encoded.
Our current glass-to-glass latency measurement is egregiously wrong because it doesn't encapsulate broadcast encoding or network latency. This timestamp would correctly capture the ~500ms of additional latency that we can measure in the lab but can't measure in production.
Synchronization would also be better, as currently two broadcasters with different encoding delays (ex. x264 and nvenc) or network RTTs are currently desynchronized.
@pkviet
i'm unearthing this PR.
Oh, that's lovely to hear! 💃
What would be your intents with it?
To be honest, you tell me ☺️ I'm more than happy to rebase + work on suggestions (eg automation around leap seconds) as long as the scope of it stays pretty much where it is right now. Also happy to hand it over to anyone else from the OBS community, if that's preferred.
What would tremendously increase the interest in the PR is if syncing of obs Media Sources was implemented [...]
I completely agree with this; unfortunately I won't find the time to add this to OBS at the moment.
@pkviet
i'm unearthing this PR.
Oh, that's lovely to hear! 💃
What would be your intents with it?
To be honest, you tell me ☺️ I'm more than happy to rebase + work on suggestions (eg automation around leap seconds) as long as the scope of it stays pretty much where it is right now. Also happy to hand it over to anyone else from the OBS community, if that's preferred.
What would tremendously increase the interest in the PR is if syncing of obs Media Sources was implemented [...]
I completely agree with this; unfortunately I won't find the time to add this to OBS at the moment.
too bad. Anyway, yeah feel free to answer the comments and rebase.
I'd like to get your feedback on an idea for handling TAI and how to integrate this (as seemlessly as possible for users + devs) into OBS.
Essentially, TAI only became a thing after MISB 0604.4 (it used to be UTC before that).
We could create this as a two-step process in the UI:
- [ ] Use MISB < 0604.4 (UTC)
- [ ] Use MISB >= 0604.4 (MISP time)
The first option should be working on most/all modern systems. It is very simple and requires nothing else but enabling it.
The second option comes with a numeric textfield (labelled "difference between UTC and TAI") + a button next to it (labelled "auto detect"). The text field is set to 0 by default and users are free to put in anything they like (within bounds). Clicking the "auto detect" button would use any method (or suitable combination) we come up with and populate the textfield, if successful.
(Potentially showing a popup on failure - might also show a popup if successful to elaborate on where we got the information from; details to be worked out).
The reason for this two step setup is twofold:
-
We have seen at least one vendor who's specifically inserting MISB 0604.2 timestamps into their streams. The playloads of the two time systems (MISB <|>= 0604.4) are indistinguishable from each other. Thus, if a user wants to synchronise OBS with a device that uses MISB < 0604.4, they would need a way to take the 8.000082 seconds between TAI + MISP time off (in addition to setting the UTC/TAI offset to
0). -
The PR in its current state (pending a rebase) would work (almost) out of the box for MISB < 0604.4. This means that the technical part of inserting the timestamps could be reviewed in isolation. Enabling MISB >= 0604.4 could then be added separately (and would maybe require different people to review).
A few more thoughts ...
I would give users the option to manually override whatever we auto-detect as the "correct"/current UTC/TAI offset. If the synchronisation between devices is off by a whole second or more, people will notice this. This might encourage them to start reading up on TAI + get an idea of what to do (+/- whole seconds are also easy to eyeball-correct by a simple try + error approach). A possible scenario for this in my head is trying to synchronise with a device that is hardcoded to a 36 seconds UTC/TAI offset (or anything else). An easy way of aligning everything could be choosing the same (wrong) offset in OBS.
An alternative mode for TAI offset handling could be that we start auto-detection when enabling the MISB >= 0604.4 option. If we fail, we fall back to the MISB < 0604.4 option. This requires some consideration around restarting OBS though.
A closing thought: splitting this topic into two PRs does not mean that I'd only do the first part and then vanish; we're very interested in getting MISB >= 0604.4 into OBS! And personally, I'd love to see this feature through myself, if I can ☺️ However, the two parts are code-technically rather distinct and smaller (fosussed) PRs are usually easier to review + get merged.
Please let me know what you think @pkviet @kixelated @derrod @notr1ch ...
If only one vendor is using UTC instead of TAI, I wouldn't bother. Thinking of UX, I'm not in favour of exposing too much stuff. The scenario I have in mind is people wanting to have synced streams, streaming from obs to obs Media Source. That' ll be the biggest population among our users; not particular broadcast vendors. For such a population which has no idea what TAI is, or what's the difference between TAI & UTC, we want to expose the minimum stuff, like a checkbox for 'NTP stream lock' even though it's actually MISP without entering into minute details of the implementation.
-
I checked the code in misp-precision-timestamp.c against the spec and it's indeed consistent with what the spec says for Precision Time Stamps (in microseconds) which work both for h264 & hevc. I didn't test against ffmpeg yet but plans to do it.
-
Rather than deferring the implementation for nvenc to later, i'd very much like to have it in this PR both for h264 and hevc, especially if you have already coded it for nvenc h264 (should be almost the same for nvenc hevc, just requires to change the 16 bytes identifier).
-
The other standard used for timecodes is SMPTE ST-12; it is incorporated into the MISP spec as 'Commercial Time Stamps'; it's not as good as MISP since it is not guaranteed to rely on absolute wallclock (NTP). I was wondering if it would be advantageous to implement it in addition to MISP, since there's guidance on converting from one system to the other (SMPTE EG 40:2016). But on second thoughts, I think it's enough to support MISP sending. It's more on the receiving side (our Media Source or VLC Source) that support for both MISP and SMPTE ST-12 is important.
We have seen at least one vendor who's specifically inserting MISB 0604.2 timestamps into their streams. The playloads of the two time systems (MISB <|>= 0604.4) are indistinguishable from each other. Thus, if a user wants to synchronise OBS with a device that uses MISB < 0604.4, they would need a way to take the 8.000082 seconds between TAI + MISP time off (in addition to setting the UTC/TAI offset to 0).
That seems like a major flaw with the standard. The client and server need to be pre-configured to use the same draft version due to the identical wire format.
This might be fine for broadcast studios that own both the sender and receiver. But for a generic ingest endpoint like Twitch and Amazon IVS, we would need to expose a way to configure the version deep in the broadcast settings. The wrong setting would work most of the time, but it would report confusing latency metrics or possibly cause desynchronization.
We would instead try to detect the version on the server side, assuming that any received timestamps <38 seconds in the future are probably using TAI/MISP instead of UTC. This gets problematic if the broadcaster has significant encoding latency or clock skew, although 38 seconds is quite a bit of buffer room.
I'd prefer a better standard but we can still work with this.
If it's not practical to automatically detect the TAI offset on every platform, then I would just use UTC.
Requiring the broadcaster to configure the UTC to MISP offset is prone to configuration mistakes. It would also cause a desync if the server automatically adjusts the offset after a leap second until the broadcaster fixes their shit. Ironically, better support for leap seconds seems like the major advantage of TAI, but it would be worse than UTC in this situation.
So yeah, I think OBS should use UTC by default, which means MISB < 0604.4. It's possible to include an option to use a newer version of the standard, but only for the sake of compatibility, as a manually configured TAI offset will just cause headaches.
So yeah, I think OBS should use UTC by default, which means MISB < 0604.4. It's possible to include an option to use a newer version of the standard, but only for the sake of compatibility, as a manually configured TAI offset will just cause headaches.
i don't like the idea of not sticking to a spec; this is killing interop. If the said vendor has complaints from users saying it's out of sync with obs, that could convince them to update to later MISP spec.
@cherusker added the 'Seeking Tester' tag. Next time you force push, it will make a build in CI; this will make testing easier.
So yeah, I think OBS should use UTC by default, which means MISB < 0604.4. It's possible to include an option to use a newer version of the standard, but only for the sake of compatibility, as a manually configured TAI offset will just cause headaches.
i don't like the idea of not sticking to a spec; this is killing interop. If the said vendor has complaints from users saying it's out of sync with obs, that could convince them to update to later MISP spec.
It's fine to use an old version of a spec. For example, HTTP/1 versus HTTP/2 and HTTP/3. Good standards will identify the version on the wire format so you can simultaneously support multiple versions.
In this case, MISB 0603.4 is objectively a better standard because it deals with leap seconds. However, if it's not possible to calculate TAI on Windows, then I think downgrading to MISB 0603.2 is perfectly acceptable. If both drafts are truely identical on the wire then that's a flaw with the standard.
Twitch is still going to have to detect/support 0603.2 even if OBS is using 0603.4, or vice versa.
To throw another wrench into the works, there will be a vote next year to potentially get rid of leap-seconds, which may result in UTC and IAT becoming the same thing, fun! https://spectrum.ieee.org/the-eight-year-leap-second-delay-might-not-be-as-bad-as-it-seems
@cherusker added the 'Seeking Tester' tag. Next time you force push, it will make a build in CI; this will make testing easier.
@pkviet thanks a lot for this! I've just rebased the PR + waiting now to see where CI takes us! 🤗
Also thanks everyone for the active participation in this thread; I'll comment shortly!
I checked the code in misp-precision-timestamp.c against the spec [...]
Thanks a lot!
[...] i'd very much like to have it in this PR both for h264 and hevc [...]
Oh yes for sure, that's not an issue at all, coming with the next push!
The other standard used for timecodes is SMPTE ST-12 [...]
This is definitely a way worth exploring. Comes with the mentioned downside however, which is the reason why my/our suggestion was to go with MISP.
That seems like a major flaw with the standard. [...]
I concur with everything you said there.
If it's not practical to automatically detect the TAI offset on every platform, then I would just use UTC.
[...] I think downgrading to MISB 0603.2 is perfectly acceptable [...]
In terms of practicality I liked @derrod's suggestion to use https://www.ietf.org/timezones/data/leap-seconds.list as that would be a simple HTTP request and some trivial text parsing on top. This does however come with the downside of potentially failing which would then need some sort of fallback (which could be using UTC instead of MISP time).
Hence why my suggestion last week to build this PR back a tiny bit and remove the TAI/MISP calculations for now. Essentially implementing MISB < 0604.4 (plain old, well-understood, and well-supported UTC). This would immediately enable downstream players to synchronise (with) outgoing OBS streams and the UI would only need a single checkbox (roughly worded "Insert MISB < 0604.4 timestamps") which would never fail.
If this sounds interesting to you @pkviet I would afterwards still love to continue this conversation however, to understand whether it's practical and advantageous for OBS (and/or downstream players of OBS) to additionally also support MISB >= 0604.4 and if it is, how that would work best in terms of UX.
Twitch is still going to have to detect/support 0603.2 even if OBS is using 0603.4, or vice versa.
The same is true for us. Thus, ultimatley not very fussed which way this goes. From a practical perspective, a (simpler) UTC approach sounds like a very promising start; esp. as it is "failproof" (for the forseeable future).
Also, wohoo, lots of green ticks from the CI! 🎉
In the interest of getting this PR either on the docket to be merged or closed, could someone clear up the following points:
- Are there existing consumers (that is ingests or technologies) that require this feature?
- What spec/requirements to these consumers have?
- Does the PR fulfil these requirements?
From the discussions it seems to me that there is only a consensus that syncing based on frame timestamps has benefits, but there is no consensus on which type of timestamp to use and there aren't actually existing implementations for it.
OBS should not be the driver for a standard here, but the consumers/users that have the need for the timestamps need to agree on what they need and only then should OBS implement that agreed-upon standard.
To answer some of your questions from the perspectives from IRL-streaming.
* Are there existing consumers (that is ingests or technologies) that require this feature?
Timecodes are for most people and services an optional feature, but for certain end-users, adding a time-code would greatly help with more complex productions where one have multiple media sources congregating. For example having two or more OBS instances in a large venue both connected to the network trough wireless or celluar connections, sending the feed to a main OBS-instance who creates the end product. Having timecodes stamped and emitted by OBS would greatly increase the production quality since the feeds would be in sync.
Timecodes is a two-part system, where it needs both the system to "stamp" the timecode, and then the system on the other end who can read the timecodes and synchronize the feeds.
I have no knowledge about MISP timestamps, as in the IRL-broadcasting world most people are referring to SEI timecodes. I dont have a preference but i am trying to collect information about the different protocols.
* What spec/requirements to these consumers have?
Most of the "prosumer" applications that is available for streamers do not support timecodes, most are arguing that its no point to implement because applications like OBS dont support reading and syncing them. So this is a classic Chicken and Egg scenario.
Professional broadcasting systems do support timecodes, but they are living in their own bubbles so the protocols and formats they use are (for me anyways) unknown. I also think that the interest for the professional broadcasting systems to get their formats supported in OBS isnt exactly on the top of their lists since syncing videofeeds often are a selling points for their "contact-us" priced in-house servers.
* Does the PR fulfil these requirements?
Thats hard to say. I dont know of any video-feed-emitting applications that use the MISP format, but since it seems to be an open standard, its defiitely doable to implement.
However it does look like this PR is purely for the "stamping" of the timecodes, and that there will still be no support to add multiple feeds into one scene which then gets synced with the timecodes. (For a full feature-cycle in OBS i assume one would need implementation in the decoder as well?)
From the discussions it seems to me that there is only a consensus that syncing based on frame timestamps has benefits, but there is no consensus on which type of timestamp to use and there aren't actually existing implementations for it.
People know what they want, but they dont know how they want it. But by principle a timecode embedded in the video-frames that contains a timecode based on Real Time Clock seems to be the consensus. Exactly how it is embedded isnt as important as the services would definitely be interested in adapting their own applications to accomodate this new feature.
OBS should not be the driver for a standard here, but the consumers/users that have the need for the timestamps need to agree on what they need and only then should OBS implement that agreed-upon standard.
I somewhat disagree here. This is a Catch-22 where it would be easier to get a consenus in OBS, than it would be to get a consensus against competing applications and consumers. Im not saying OBS should create their own standard, but implementig any open standard at all would really benefit consumers since they then can implement it themselves after OBS starts supporting it.
With that said, i want to invite any consumers/users that has an interest in timecodes and synced feeds to contribute (PR, Issues, DM me, etc) to this project where i collect information about the different formats, statuses and ultimately instructions on how to implement a common format among the consumers.
