CycleGAN-TensorFlow icon indicating copy to clipboard operation
CycleGAN-TensorFlow copied to clipboard

Published 20 hours ago • verified publisher icon vanhuyz

Training doesn't start

Open philthestone opened this issue 3 years ago • 1 comments

Hi there, Thank you for the tensorflow impelementation!

I've run into the following problem. I am working on google colab and somehow the training is not starting, without giving a clear %tensorflow_version 1.x

I'll initialize training with:

!python /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py
--X=/content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/VGS4_invivo/invivo4.tfrecords
--Y=/content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/VGS1_invivo/invivo1.tfrecords
--image_size=1024

I'll get the follwing output:

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:135: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:135: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:49: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

W0715 09:01:16.252792 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:49: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:58: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0715 09:01:16.255107 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:58: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:19: TFRecordReader.init (from tensorflow.python.ops.io_ops) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.TFRecordDataset. W0715 09:01:16.256774 140252217407360 deprecation.py:323] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:19: TFRecordReader.init (from tensorflow.python.ops.io_ops) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.TFRecordDataset. WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:28: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). W0715 09:01:16.258310 140252217407360 deprecation.py:323] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:28: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:277: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). W0715 09:01:16.264339 140252217407360 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:277: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs). If shuffle=False, omit the .shuffle(...). WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:189: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensors(tensor).repeat(num_epochs). W0715 09:01:16.265325 140252217407360 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:189: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.from_tensors(tensor).repeat(num_epochs). WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:198: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. W0715 09:01:16.267258 140252217407360 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:198: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:198: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. W0715 09:01:16.268559 140252217407360 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/input.py:198: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:32: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W0715 09:01:16.274218 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:32: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:35: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0715 09:01:16.274413 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:35: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:52: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W0715 09:01:16.278169 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:52: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:45: shuffle_batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size). W0715 09:01:16.286282 140252217407360 deprecation.py:323] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:45: shuffle_batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.shuffle(min_after_dequeue).batch(batch_size). WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:48: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0715 09:01:16.298265 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/reader.py:48: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:21: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0715 09:01:16.336498 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:21: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:185: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W0715 09:01:16.337138 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:185: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:188: calling RandomNormal.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0715 09:01:16.337533 140252217407360 deprecation.py:506] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:188: calling RandomNormal.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:229: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

W0715 09:01:16.365534 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/ops.py:229: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:50: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

W0715 09:01:17.149662 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:50: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:50: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0715 09:01:17.150010 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/generator.py:50: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:168: The name tf.squared_difference is deprecated. Please use tf.math.squared_difference instead.

W0715 09:01:18.426850 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:168: The name tf.squared_difference is deprecated. Please use tf.math.squared_difference instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:87: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0715 09:01:18.930492 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:87: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:92: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0715 09:01:19.552722 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:92: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:119: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

W0715 09:01:20.892452 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:119: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:122: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0715 09:01:20.899345 140252217407360 deprecation.py:323] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:122: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:129: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

W0715 09:01:20.901338 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/model.py:129: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:67: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

W0715 09:01:31.051686 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:67: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:68: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0715 09:01:31.053035 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:68: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:69: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0715 09:01:33.183943 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:69: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:71: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0715 09:01:33.523708 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:71: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-07-15 09:01:33.524848: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-07-15 09:01:33.539370: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.539987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2021-07-15 09:01:33.540303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2021-07-15 09:01:33.541814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-07-15 09:01:33.543457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2021-07-15 09:01:33.543788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2021-07-15 09:01:33.545302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2021-07-15 09:01:33.546091: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2021-07-15 09:01:33.549201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-07-15 09:01:33.549317: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.549956: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.550518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-07-15 09:01:33.550839: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F 2021-07-15 09:01:33.555161: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000175000 Hz 2021-07-15 09:01:33.555339: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555c569c2bc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-07-15 09:01:33.555366: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-07-15 09:01:33.641587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.642394: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555c569c2d80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-07-15 09:01:33.642428: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2021-07-15 09:01:33.642622: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.643241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2021-07-15 09:01:33.643306: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2021-07-15 09:01:33.643327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-07-15 09:01:33.643346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2021-07-15 09:01:33.643365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2021-07-15 09:01:33.643383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2021-07-15 09:01:33.643399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2021-07-15 09:01:33.643417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-07-15 09:01:33.643490: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.644073: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.644582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-07-15 09:01:33.644661: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2021-07-15 09:01:33.645681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-15 09:01:33.645706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-07-15 09:01:33.645716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-07-15 09:01:33.645829: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.646432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 09:01:33.646948: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0. 2021-07-15 09:01:33.647001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:79: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

W0715 09:01:33.647766 140252217407360 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:79: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:83: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. W0715 09:01:35.981049 140252217407360 deprecation.py:323] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-TensorFlow-master/train.py:83: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module.

After that, no further outputs, no error messages. But the training doesn't start either. Any ideas how to solve this problem? Thank a lot

philthestone avatar Jul 15 '21 09:07 philthestone

guess that you should use tensorflow 1.x version

Lenoids avatar Oct 11 '21 13:10 Lenoids