Improve handling of rates when converting to timecode strings

[jminor]

trafficstars

This PR improves OTIO's to_timecode, is_valid_timecode_rate, and nearest_valid_timecode_rate functions so they handle rate values which are close, but not exactly matching the correct rates.

Specifically the previous strategy of keeping a table of commonly mis-typed non-integer rates (e.g. 29.97, 29.976, 23.98, etc.) is replaced by a heuristic which matches rates to the closest correct rate within some tolerance.

Rates checked by is_valid_timecode_rate and nearest_valid_timecode_rate now match each other, and adhere to ST 12-1:2014 - SMPTE Standard - Time and Control Code (which is now freely available 👏🎉)

We also spot-checked some comparisons between Avid Media Composer and OTIO to make sure we got the drop frames in the right spot - that was already correct before this PR.

Note: there is 1 test failing which needs to be addressed, and my updates to the other tests should be scrutinized to make sure the changes are good.

For the failing test... something doesn't match up.

• Avid Media Composer's Calculator says: 1084319 frames = 05:01:11:29 @ 60 fps
• OpenTimelineIO says: 1084319 frames = 05:01:11:59 @ 60 fps
• https://robwomack.com/timecode-calculator/ says: 1084319 frames = 05:01:30:03 @ 60 fps

Test code: otio.opentime.from_frames(1084319, 60).to_timecode()

[meshula]

To summarize, MC and OTIO ~~agree~~ disagree, but time-code-calculator ~~significantly differs~~ agrees with OTIO. ~~?~~ Maybe we need to ping Avid. ~~the calculator's author...~~

[markreidvfx]

I would like to suggest RationalTime::from_timecode explicitly take the nominal fps as a argument along with the frame rate

like what I proposed here: https://github.com/AcademySoftwareFoundation/OpenTimelineIO/issues/1452#issuecomment-1278486164

RationalTime
RationalTime::from_timecode(std::string const& timecode, 
                            uint32_t           timecode_fps,
                            double             rate,
                            ErrorStatus*       error_status)

This would eliminate the need to do all the guessing from the playback frame rate.  You might for example be using 24 frame timecode at playback rate of 30 fps, which isn't possible with the current method.

[markreidvfx]

I tried that online tc calculator. Its pretty finicky but you might have had it set to 59.97fps DF , I get 1084319 frames = 05:01:11:59 @ 60 fps.
1084319 frames = 05:01:11:59 @ 59.97 fps. Non Drop Frame (should be the same as 60) and 1084319 frames = 05:01;30;03 @ 59.97 fps Drop Frame

That rate should really be 59.94, not 59.97, but oh well. I haven't figure out how to get 59.94 DF TC working on my avid yet to verify if that calculation is correct.

[jminor]

MC and OTIO are off by 30 frames from each other.

[markreidvfx]

The TC display in MC is might be showing 30fps TC where it duplicates every step. My MC cannot display 60 TC without using a burn in.

To make things a little more odd, the 60fps TC burnin only seems to works when working at 59.94fps. I am still using 2018, maybe there is better support in a newer version.

[markreidvfx]

ok, in MC general settings there is option to change the TC display format from 30fps to 60 fps

Still haven't found a 60fps drop frame option.

[jminor]

Here's the Media Composer features I used to check against:

• The "=" menu in the bottom left corner of the timeline window has "Show Track" which lets me turn on multiple timecode rulers along the timeline:

• The Timecode Window right-click menu lets me choose these:

• The Calculator Window lets you convert between "Total Frame Count" and various rates. This is where I got 1084319 frames = 05:01:11:29 @ 60 fps

[jminor]

@markreidvfx can you explain more about using 24 frame timecode with 30 fps playback? That's not a scenario I've seen, so I'm confused about how it works.

With your proposed change, would these produce the same answer?

a = otio.opentime.from_timecode(timecode_str, 24, 30)
b = otio.opentime.from_timecode(timecode_str, 24, 24).rescaled_to(30)

and the other direction:

a = otio.opentime.from_frames(frame_number, 30).rescaled_to(24).to_timecode()
b = otio.opentime.from_frames(frame_number, 30).to_timecode(rate=24)

[markreidvfx]

No, those would produce different answers. maybe using 23.97 is a bit more clear. It would be the equivalent of this

frame = otio.opentime.from_timecode(timecode_str, 24).value
a = otio.opentime.from_frames(frame, 23.97)
b = otio.opentime.from_timecode(timecode_str, 24).rescaled_to(23.97)

a.to_seconds() != b.to_seconds()

Timecode is converted to a frame number first. Then that frame number is converted to a time at current frame rate.

Maybe adding another arg isn't the correct answer. The big thing I'm trying to get across is that timecode should be treated as frame number encoding and is not always reliable to be a timestamp. It should really be called framecode in my opinion. :p

This is why I normally use a pattern like this

frames_to_seconds(timecode_to_frames(timecode_str, timecode_fps), frame_rate)

and the other direction:

frames_to_timecode(seconds_to_frames(seconds, frame_rate), timecode_fps, drop=False)

[markreidvfx]

what is the the project format/edit rate?

some formats can't access certain timecodes.

[jminor]

Aha! That explanation makes total sense now. The conversion to a frame count vs a timestamp really makes it clear - thanks @markreidvfx !

I'll check my MC project settings tomorrow.

I'd love to compare this to Resolve and/or Premiere as well. Are there other trusted timecode conversion tools or software libraries out there to compare to?

[jminor]

This PR overlaps or conflicts with this older PR: https://github.com/AcademySoftwareFoundation/OpenTimelineIO/pull/1180

[meshula]

@jminor I closed that one in favor of this one, since this one rectifies the treatments of documented SMPTE rates with the code. @splidje NLEs have various "tricks" for dealing with high frame rates, let's carry on the discussion of what to do with the high frame rates here.

[markreidvfx]

I've verified those calculations in Resolve, it actually supports 59.94 TC with drop frame.

1084319 frames = 05:01:11:59 @ 59.94 fps without drop frame 1084319 frames = 05:01;30;03 @ 59.94 fps with Drop Frame

You can set the tc format when you create a timeline or in your project settings

You can toggle between frames and tc by right clicking on the timecode in the viewer

[markreidvfx]

Resolve also additionally supports a 16, 18, 47.952 and 48 TC format. The 47.952 appears to just be 48, like 23.976 is just 24

[meshula]

Thanks for doing the extra checks! So I think the theory is that MC is displaying time code at 30, even for a selected rate of 60 (as opposed to MC displaying a value that is 30 frames off of our calculation). A check for that would be to knock off 5 frames at 30, and check if MC then shows 24 instead of 29, or if it shows 19. 24 would demonstrate it is displaying a 30 rate, instead of 60; whereas 19 would show that the calculation is off by 30.

[markreidvfx]

I'm quite sure @jminor had his avid set in 30 TC mode for 60. Its the default setting. The timecode ticks every 2 frames and the top dot of the last colon blinks on and off to tell you if your on an even or odd frame.

https://user-images.githubusercontent.com/814966/203416726-a1cbd530-0c0e-4073-993f-79702d81e35e.mp4

This mode is somewhat described in the spec intro, page 5

Progressive video systems with frame rates above 30 frames per second are described in this document, documenting what have become “de facto” implementations. Since the frame rate of these 50 and 60 frames-per-second progressive systems exceeds the frame count capacity of the time address, counting is done on frame pairs, which results in an edit resolution of two frames using traditional linear time code.

There are some further details in 12.1

[markreidvfx]

I've also verified those timecode values in Unreal Engine

60 fps NDF

59.94 fps DF

[jminor]

Sorry for the delay in getting back to this. I should hopefully have time to revisit this in a couple of weeks.

In the meantime, I happened across this which is interesting to compare/contrast with OTIO's approach: https://github.com/orchetect/TimecodeKit

[meshula]

Had a look at the link you provided, quite an interesting read ~ TimeCodeKit's major difference, I'd say, is that it introduces a strongly typed TimeCode object from which a string can be generated, and which functions as a mathematical object. It hasn't got a stronger ability than otio::RationalTime to represent a point in time, or a stronger ability to do math than otio::RationalTime, and would probably resist computations we are interested in, such as linear timewarps.

Another point of contrast, philosophically, is the view suggested by @markreidvfx earlier in the thread, that the timecode string should be considered, in a way, a "rendered" interpretation of a time, as opposed to a ground truth representation of time.

TimeCodeKit does some interesting things that we don't support, at all. If you decrement time below zero, it will wrap around to 24 hours. We generate negative values. We could possibly do something similar in genarating time code strings. At the moment, negative time values are rejected. If the are negative and greater than -24 hours, we could wrap around.

I think it would be worth working through the math of TimeCodeKit to determine whether our conversion algorithm matches. Also, it would be worth working through the unit tests, and perhaps adding one to one corresponding unit tests to our own code.

[meshula]

I ran the tests in my own repo, the python unit tests are what failed, the tests themselves may need correction.

======================================================================
FAIL: test_invalid_rate_to_timecode_functions (test_opentime.TestTime.test_invalid_rate_to_timecode_functions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/OpenTimelineIO/OpenTimelineIO/tests/test_opentime.py", line 400, in test_invalid_rate_to_timecode_functions
    with self.assertRaises(ValueError):
AssertionError: ValueError not raised

======================================================================
FAIL: test_timecode_infer_drop_frame (test_opentime.TestTime.test_timecode_infer_drop_frame)
---------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/OpenTimelineIO/OpenTimelineIO/tests/test_opentime.py", line 347, in test_timecode_infer_drop_frame
    self.assertEqual(t.to_timecode(rate, drop_frame=None), timecode)
AssertionError: '05:01:30;03' != '05:01:11;59'
- 05:01:30;03
+ 05:01:11;59

[jminor]

Yes, @meshula that failing test is the one I mentioned in the initial post at the very top of this thread. My question is/was: what is the correct return value for otio.opentime.from_frames(1084319, 60).to_timecode() ?

[meshula]

otio.opentime.from_frames(1084319, 60).to_timecode() # no dropframe rate requested

referencing @markreidvfx's reported values from Resolve and Unreal

1084319 frames = 05:01:11:59 @ 59.94 fps without drop frame 1084319 frames = 05:01:30;03 @ 59.94 fps with Drop Frame

suggest that answer should be 05:01:11:59

I've gone through this thread a bunch of times in the last hour, and I feel we can be confident about that.

[jminor]

Following some illuminating conversation with @markreidvfx and Roger @ Avid, I have a better understanding of how the current OTIO timecode API is misleading. See for example this:

>>> otio.opentime.from_frames(1, 3).to_timecode(30)
'00:00:00:10'
>>> otio.opentime.from_frames(1, 30).to_timecode(30)
'00:00:00:01'
>>> otio.opentime.from_frames(1, 300).to_timecode(30)
'00:00:00:00'
>>> otio.opentime.from_frames(1, 3).to_timecode(30000/1001, True)
'00:00:00;09'
>>> otio.opentime.from_frames(1, 30).to_timecode(30000/1001, True)
'00:00:00;00'
>>> otio.opentime.from_frames(1, 300).to_timecode(30000/1001, True)
'00:00:00;00'

[meshula]

Could you clarify why it's misleading? Maybe I've been looking at it too long and have got some assumptions that make me overlook something.

otio.opentime.from_frames(1, 3).to_timecode(30) '00:00:00:10'

1 frame 3fps, render to 30 fps, 10 frames.

otio.opentime.from_frames(1, 30).to_timecode(30) '00:00:00:01'

1 @ 30fps to 30 = 1

otio.opentime.from_frames(1, 300).to_timecode(30) '00:00:00:00'

1 @ 300fps to 30 is 0.1, round to zero

I don't have intuition about drop frame and haven't pulled out a calculator, but if I squint maybe these are fine?

otio.opentime.from_frames(1, 3).to_timecode(30000/1001, True) '00:00:00;09' otio.opentime.from_frames(1, 30).to_timecode(30000/1001, True) '00:00:00;00' otio.opentime.from_frames(1, 300).to_timecode(30000/1001, True) '00:00:00;00'

[reinecke]

Some relevant links from the TSC meeting:

• From @jhodges10 Some folks at Frame.io published a paper with SMPTE re: timecode handling that runs over 24 hours (alternate link)
• From Ben Schofield: Timecode in IMF paper

[meshula]

Hi! One year later, are we stalemated? Today I am wondering if the work here fixes the very sad rounding bug reported yesterday.

How can we unstick this one? Have we bikeshedded it to death? Is this PR demonstrably an incremental improvement in what we have? If it is not perfect but incrementally better, and not incrementally worse, I want to merge it.

I am lost in the weeds of the many comment threads here. Please send help, SOS.

[jminor]

Re-reading the thread, it looks like I need to do these things:

• Rebase this onto main
• Use ~~"05:01:11:59"~~ "05:01:30;03" for the failing test.
• Rename functions to use "smpte" instead of the word "valid"

For the method rename, do we need to deprecate the old names & support both for a release?

[apetrynet]

The use of the "valid_xx" functions have come up in quite a few discussions and issues. I think we should supply both new and old function names with a deprecation warning in the old one for at least the upcoming release.

[meshula]

@apetrynet I'm not following the comment "The use of the "valid_xx" functions have come up in quite a few discussions and issues." are you saying that supports merging, or that there some other work?

@jminor I concur with @apetrynet on the new/old+deprecation strategy,

Thanks for looking at this, folks :D