Error in Bayesian Optimization

[MichailChatzianastasis]

Hey, I got the following error when i try to run bayesian optimization in ENAS. Any ideas how to fix this?

2020-08-27 09:45:05.767860: E tensorflow/core/common_runtime/executor.cc:623] Executor failed to create kernel. Invalid argument: Default AvgPoolingOp only supports NHWC on device type CPU [[{{node child_1/layer_1/pool_at_1/from_0/AvgPool}} = AvgPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] Traceback (most recent call last): File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: Default AvgPoolingOp only supports NHWC on device type CPU [[{{node child_1/layer_1/pool_at_1/from_0/AvgPool}} = AvgPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "bo.py", line 268, in score = -eva.eval(arc) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/evaluation.py", line 260, in eval return self.ops["eval_func"](self.sess, "valid", feed_dict={self.ops["controller"]["sample_arc3"]: np.asarray(arch_str)}) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py", line 715, in customized_eval_once acc = self.eval_once(sess, eval_set, feed_dict, verbose) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/models.py", line 179, in eval_once acc = sess.run(acc_op, feed_dict=feed_dict) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run run_metadata=run_metadata) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run raise six.reraise(*original_exc_info) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1240, in run return self._sess.run(*args, **kwargs) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1312, in run run_metadata=run_metadata) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run return self._sess.run(*args, **kwargs) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Default AvgPoolingOp only supports NHWC on device type CPU [[node child_1/layer_1/pool_at_1/from_0/AvgPool (defined at /home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py:145) = AvgPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'child_1/layer_1/pool_at_1/from_0/AvgPool', defined at: File "bo.py", line 130, in eva = Eval_NN() # build the network acc evaluater File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/evaluation.py", line 304, in Eval_NN eva = Eval() File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/evaluation.py", line 234, in init self.ops = self.get_ops(images, labels) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/evaluation.py", line 178, in get_ops child_model.connect_controller(controller_model) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py", line 736, in connect_controller self._build_valid() File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py", line 648, in _build_valid logits = self._model(self.x_valid, False, reuse=True) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py", line 222, in _model layer, out_filters, 2, is_training) File "/home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py", line 145, in _factorized_reduction x, [1, 1, 1, 1], stride_spec, "VALID", data_format=self.data_format) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 2110, in avg_pool name=name) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 72, in avg_pool data_format=data_format, name=name) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(*args, **kwargs) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/anaconda/envs/azureml_py36_automl/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Default AvgPoolingOp only supports NHWC on device type CPU [[node child_1/layer_1/pool_at_1/from_0/AvgPool (defined at /home/mchatzi/my_projects/D-VAE/bayesian_optimization/../software/enas/src/cifar10/eval_child.py:145) = AvgPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[muhanzhang]

From the error message, did you use CPU instead of GPU? Using CPU will be very slow. And CPU only supports NHWC format images while the pretrained ENAS model was using GPU that relied on NCHW format.

[MichailChatzianastasis]

Thanks for your reply. I followed the instructions in the readme. Should i change something in order to use GPU?

[muhanzhang]

If your machine has a GPU and you install TensorFlow's GPU version, then the program should automatically use GPU. You can also check your GPU usage by "watch nvidia-smi", to verify whether GPU is really used.