CueGUI/Cuebot: job.getUpdatedFrames(lastCheck) hangs if lastCheck is > 60sec old

[angelali-ms]

Describe the bug Currently, there is a timestamp discrepancy between our Cuebot Server (Linux) and our artist workstations (Windows). The Cuebot Server is 50 seconds faster than the artist workstations.

This causes us to consistently hit the edge case where the getUpdatedFrames() call throws a "timestamp cannot be over a minute off" exception when we have CueGUI open.

However, the job.getUpdatedFrames() call does not throw an opencue.exception.CueException (as expected in CueGUI). Instead, it hangs until it hits the Cuebot.Timeout and throws a grpc._channel._InactiveRpcError error.

To Reproduce To reproduce the hang and the timeout:

1. Configure the cuebot.timeout value in the opencue.yaml/default.yaml file to a value that's smaller (I set it to 10 for 10 seconds)
2. Start up a Python shell and run the following

import opencue
import time
jobs = opencue.api.getJobs(include_finished=True)
j = jobs[0]
last_check = int(time.time()) - 100  # forcing the timestamp difference to be over 60 seconds
j.getUpdatedFrames(last_check)

3. The getUpdatedFrames call should hang for cuebot.timeout seconds and return a traceback similar to this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\users\angela.li\dev\farm\opencue\pycue\opencue\wrappers\job.py", line 215, in getUpdatedFrames
    timeout=Cuebot.Timeout)
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_interceptor.py", line 221, in __call__
    compression=compression)
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_interceptor.py", line 257, in _with_call
    return call.result(), call
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_channel.py", line 343, in result
    raise self
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_interceptor.py", line 247, in continuation
    compression=new_compression)
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_channel.py", line 957, in with_call
    return _end_unary_response_blocking(state, call, True, None)
  File "C:\Users\angela.li\.virtualenvs\OpenCue-bR_PhmGX\lib\site-packages\grpc\_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.DEADLINE_EXCEEDED
        details = "Deadline Exceeded"
        debug_error_string = "{"created":"@1708128221.322000000","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}"
>

Expected behavior CueGUI is expecting the getUpdatedFrames() call to immediately throw an opencue.exception.CueException.

As a hack in my local version, I've decreased the cuebot.timeout value and added a catch for grpc._channel._InactiveRpcError that returns None so it forces a full update when the call times out. Don't think this is the proper solution, but I'm happy to create a pull request for this if needed.

Version Number 0.22 / commit: a694b8dce53e6adf804abfe6a758316946e87124

Additional context Happy to provide additional context / screenshots if needed, thanks!