server
server copied to clipboard
CasparCG
Server not responsive after back dating system time
Expected behaviour
Server should continue to run irrespective of what the system date and time is.
Current behaviour
Server stops responding (frozen) when the system date or time is set to a value less than the current time. The server starts responding again when the system time gets to the point at which it was when you first changed it, but plays the video at a faster rate before becoming normal.
Steps to reproduce
- Start the server
- Add a screen consumer (eg ADD 1-99 SCREEN 0)
- Play any video (eg PLAY 1-0 amb)
- Back date the system time.
Environment
- Commit: d442a8f8
- Server version: 2.3
- Operating system: Windows 10
Screenshots
If applicable, add screenshots as complementary information.
A correct NTP implementation (ntp.org, chrony, meinberg, ...) adjusts the clock speed and does not make the clock jump.
I think this is down to the channel tick clock in your setup being done based on the system clock, so I can understand changing it to cause that (it would be waiting until it passed the original time to produce the next frame). It should be possible to do something here though, to make this at most be a couple of frames of glitch instead
This problem is caused by a known Microsoft Visual Studio bug: https://developercommunity.visualstudio.com/content/problem/61684/stdthis-threadsleep-for-depends-on-system-time.html.
I've tried some different approaches and they have all ended up in the same place. I imagine spinning a CPU reading the current time could be made to work with care but that's not a great plan.
@scriptorian, can you just sum what you have found so far regarding this issue? Thanks Simon.
Further information: Any consumer which reports that it doesn't have a synchronization clock relies on core/consumer/output.cpp executing a once per frame sleep until the next frame is due to be processed. This sleep is done using the C++ std::this_thread::sleep_until function, with the 'until' time calculated to be one frame time later than the last call. If the system time is changed backwards then the wait will increase by the amount the time has been adjusted. The Microsoft standard library team have apparently fixed this but it cannot be released because it is a change that breaks binary compatibility. Please see the link above for more details. As I note there I haven't so far found a way around this problem. I assume that this problem will not arise with a linux build but I currently can't test that.
Based on the information we have on this we'll drop it from the milestone because it is dependent on a bug in Visual Studio.
I'd also say there is a problem in using high_resolution_clock instead of steady_clock, since high_resolution_clock is not guaranteed to be either steady or monotonously increasing, both of these aspects being absolutely critical to correct functioning of frame timing.
