OnionNet-2 icon indicating copy to clipboard operation
OnionNet-2 copied to clipboard

predict.py : CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

Open AntonHuu opened this issue 1 year ago • 0 comments

Hey! I have used ON2 before however now that I am trying to use it on a different computer/GPU I run into the next error with tensorflow/GPU. I was able to run the generation.py script via GPU, however for the predict.py script it crashes with the following error.

Traceback (most recent call last): File "/path//OnionNet-2/predict.py", line 68, in model = tf.keras.models.load_model(args.model, File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/saving/save.py", line 182, in load_model return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 177, in load_model_from_hdf5 model = model_config_lib.model_from_config(model_config, File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/saving/model_config.py", line 55, in model_from_config return deserialize(config, custom_objects=custom_objects) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/layers/serialization.py", line 171, in deserialize return generic_utils.deserialize_keras_object( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object return cls.from_config( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 484, in from_config model = cls(name=name) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper result = method(self, *args, **kwargs) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/engine/sequential.py", line 116, in init super(functional.Functional, self).init( # pylint: disable=bad-super-call File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper result = method(self, *args, **kwargs) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 308, in init self._init_batch_counters() File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/training/tracking/base.py", line 457, in _method_wrapper result = method(self, *args, **kwargs) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 317, in _init_batch_counters self._train_counter = variables.Variable(0, dtype='int64', aggregation=agg) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 262, in call return cls._variable_v2_call(*args, **kwargs) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 244, in _variable_v2_call return previous_getter( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 237, in previous_getter = lambda **kws: default_variable_creator_v2(None, **kws) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/variable_scope.py", line 2633, in default_variable_creator_v2 return resource_variable_ops.ResourceVariable( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/variables.py", line 264, in call return super(VariableMetaclass, cls).call(*args, **kwargs) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1507, in init self._init_from_args( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1650, in _init_from_args initial_value = ops.convert_to_tensor( File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1499, in convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/tensor_conversion_registry.py", line 52, in _default_conversion_function return constant_op.constant(value, dtype, name=name) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 263, in constant return _constant_impl(value, dtype, shape, name, verify_shape=False, File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 275, in _constant_impl return _constant_eager_impl(ctx, value, dtype, shape, verify_shape) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 300, in _constant_eager_impl t = convert_to_eager_tensor(value, ctx, dtype) File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 97, in convert_to_eager_tensor ctx.ensure_initialized() File "/path//miniconda3/envs/OnionNet-2/lib/python3.8/site-packages/tensorflow/python/eager/context.py", line 539, in ensure_initialized context_handle = pywrap_tfe.TFE_NewContext(opts) tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

AntonHuu avatar Jul 18 '24 15:07 AntonHuu