No OpKernel was registered to support Op 'PreprocessingForward' Error for Multi Machine, Multi GPU

[wangcaihua]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.04): Linux Ubuntu 20.04, Offical GPU Image 2304
• DeepRec version or commit id: deeprec2302
• Python version: 3.8.10
• Bazel version (if compiling from source): not compiling from source
• GCC/Compiler version (if compiling from source): gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
• CUDA/cuDNN version: 11.6

Describe the current behavior [1,9]:Traceback (most recent call last): [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call [1,9]: return fn(*args) [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn [1,9]: self._extend_graph() [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph [1,9]: tf_session.ExtendSession(self._session) [1,9]:tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'PreprocessingForward' used by {{node input_layer/input_layer/group_embedding_lookup/PreprocessingForward/PreprocessingForward}}with these attrs: [rank=9, id_in_local_rank=0, num_ranks=16, num_gpus=16, Toffsets=DT_INT64, Tindices=DT_INT64, num_lookups=26, combiners=["mean", "mean", "mean", "mean", "mean", ..., "mean", "mean", "mean", "mean", "mean"], dimensions=[16, 16, 16, 16, 16, ..., 16, 16, 16, 16, 16], shard=[-1, -1, -1, -1, -1, ..., -1, -1, -1, -1, -1]] [1,9]:Registered devices: [CPU, XLA_CPU] [1,9]:Registered kernels: [1,9]: device='GPU'; Tindices in [DT_INT32]; Toffsets in [DT_INT32] [1,9]: device='GPU'; Tindices in [DT_INT32]; Toffsets in [DT_INT64] [1,9]: device='GPU'; Tindices in [DT_INT64]; Toffsets in [DT_INT32] [1,9]: device='GPU'; Tindices in [DT_INT64]; Toffsets in [DT_INT64] [1,9]: [1,9]: [[input_layer/input_layer/group_embedding_lookup/PreprocessingForward/PreprocessingForward]] [1,9]: [1,9]:During handling of the above exception, another exception occurred: [1,9]: [1,9]:Traceback (most recent call last): [1,9]: File "train.py", line 887, in [1,9]: main() [1,9]: File "train.py", line 642, in main [1,9]: train(sess_config, hooks, model, train_init_op, train_steps, [1,9]: File "train.py", line 505, in train [1,9]: with tf.train.MonitoredTrainingSession( [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 655, in MonitoredTrainingSession [1,9]: return MonitoredSession( [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1085, in init [1,9]: super(MonitoredSession, self).init( [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 800, in init [1,9]: self._sess = _RecoverableSession(self._coordinated_creator) [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1282, in init [1,9]: _WrappedSession.init(self, self._create_session()) [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1287, in _create_session [1,9]: return self._sess_creator.create_session() [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 953, in create_session [1,9]: self.tf_sess = self._session_creator.create_session() [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 713, in create_session [1,9]: return self._get_session_manager().prepare_session( [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/session_manager.py", line 306, in prepare_session [1,9]: sess.run(init_op, feed_dict=init_feed_dict) [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 955, in run [1,9]: result = self._run(None, fetches, feed_dict, options_ptr, [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1179, in _run [1,9]: results = self._do_run(handle, final_targets, final_fetches, [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1358, in _do_run [1,9]: return self._do_call(_run_fn, feeds, fetches, targets, options, [1,9]: File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call [1,9]: raise type(e)(node_def, op, message) [1,9]:tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'PreprocessingForward' used by node input_layer/input_layer/group_embedding_lookup/PreprocessingF[1,9]:orward/PreprocessingForward (defined at /usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py:1748) with these attrs: [rank=9, id_in_local_rank=0, num_ranks=16, num_gpus=16, Toffsets=DT_INT64, Tindices=DT_INT64, num_lookups=26, combiners=["mean", "mean", "mean", "mean", "mean", ..., "mean", "mean", "mean", "mean", "mean"], dimensions=[16, 16, 16, 16, 16, ..., 16, 16, 16, 16, 16], shard=[-1, -1, -1, -1, -1, ..., -1, -1, -1, -1, -1]] [1,9]:Registered devices: [CPU, XLA_CPU] [1,9]:Registered kernels: [1,9]: device='GPU'; Tindices in [DT_INT32]; Toffsets in [DT_INT32] [1,9]: device='GPU'; Tindices in [DT_INT32]; Toffsets in [DT_INT64] [1,9]: device='GPU'; Tindices in [DT_INT64]; Toffsets in [DT_INT32] [1,9]: device='GPU'; Tindices in [DT_INT64]; Toffsets in [DT_INT64] [1,9]: [1,9]: [[input_layer/input_layer/group_embedding_lookup/PreprocessingForward/PreprocessingForward]]

Describe the expected behavior

Code to reproduce the issue

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.