2022-08-18 12:58:05.658194: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.659580: W ./tensorflow/stream_executor/stream.h:2119] attempting to perform DNN operation using StreamExecutor without DNN support
2022-08-18 12:58:05.691343: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.696118: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNIMPLEMENTED: DNN library is not found.
2022-08-18 12:58:05.697546: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.698416: W ./tensorflow/stream_executor/stream.h:2119] attempting to perform DNN operation using StreamExecutor without DNN support
2022-08-18 12:58:05.699909: E tensorflow/stream_executor/cuda/cuda_dnn.cc:377] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.1.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2022-08-18 12:58:05.700836: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops.cc:1120 : UNIMPLEMENTED: DNN library is not found.
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 572, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 162, in process
input_frames, _TIMES_TO_INTERPOLATE.value, self.interpolator))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 120, in interpolate_recursively_from_files
times_to_interpolate, interpolator)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 88, in _recursive_generator
np.expand_dims(frame1, axis=0), np.expand_dims(frame2, axis=0), time)[0]
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 104, in interpolate
result = self._model(inputs, training=False)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/load.py", line 686, in _call_attribute
return instance.call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 55, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:
Detected at node 'AvgPool' defined at (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 457, in get
self.data_channel_factory)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 865, in init
op.setup()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 141, in setup
_MODEL_PATH.value)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 82, in init
self._model = tf.compat.v2.saved_model.load(model_path)
Node: 'AvgPool'
dnn PoolForward launch failed
[[{{node AvgPool}}]] [Op:__inference_restored_function_body_13284]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 635, in process_bundle
bundle_processor.process_bundle(instruction_id))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 996, in process_bundle
element.data)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 221, in process_encoded
self.output(decoded_value)
File "apache_beam/runners/worker/operations.py", line 352, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 354, in apache_beam.runners.worker.operations.Operation.output
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1300, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1396, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1300, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 571, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1396, in apache_beam.runners.common._OutputProcessor.process_outputs
File "apache_beam/runners/worker/operations.py", line 216, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
File "apache_beam/runners/worker/operations.py", line 713, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/worker/operations.py", line 714, in apache_beam.runners.worker.operations.DoOperation.process
File "apache_beam/runners/common.py", line 1235, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 1316, in apache_beam.runners.common.DoFnRunner._reraise_augmented
File "apache_beam/runners/common.py", line 1233, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 572, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 162, in process
input_frames, _TIMES_TO_INTERPOLATE.value, self.interpolator))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 120, in interpolate_recursively_from_files
times_to_interpolate, interpolator)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/util.py", line 88, in _recursive_generator
np.expand_dims(frame1, axis=0), np.expand_dims(frame2, axis=0), time)[0]
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 104, in interpolate
result = self._model(inputs, training=False)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/saved_model/load.py", line 686, in _call_attribute
return instance.call(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 55, in quick_execute
inputs, attrs, num_outputs)
RuntimeError: tensorflow.python.framework.errors_impl.InternalError: Graph execution error:
Detected at node 'AvgPool' defined at (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 187, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 183, in main
_run_pipeline()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 176, in _run_pipeline
result = pipeline.run()
File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 573, in run
return self.runner.run_pipeline(self, self._options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/direct/direct_runner.py", line 131, in run_pipeline
return runner.run_pipeline(pipeline, options)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 196, in run_pipeline
pipeline.to_runner_api(default_environment=self._default_environment))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 206, in run_via_runner_api
return self.run_stages(stage_context, stages)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 385, in run_stages
runner_execution_context, bundle_context_manager)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 653, in _run_stage
bundle_manager))
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 770, in _run_bundle
data_input, data_output, input_timers, expected_timer_output)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1080, in process_bundle
result_future = self._worker_handler.control_conn.push(process_bundle_req)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 378, in push
response = self.worker.do_instruction(request)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 598, in do_instruction
getattr(request, request_type), request.instruction_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 629, in process_bundle
instruction_id, request.process_bundle_descriptor_id)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 457, in get
self.data_channel_factory)
File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 865, in init
op.setup()
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator_cli.py", line 141, in setup
_MODEL_PATH.value)
File "/usr/local/lib/python3.7/dist-packages/frame_interpolation/eval/interpolator.py", line 82, in init
self._model = tf.compat.v2.saved_model.load(model_path)
Node: 'AvgPool'
dnn PoolForward launch failed
[[{{node AvgPool}}]] [Op:__inference_restored_function_body_13284] [while running 'Process directories']
2022-08-18 12:58:07.798 | INFO | main::14 - Interpolation comlpete.