CycleGAN-tensorflow
CycleGAN-tensorflow copied to clipboard
Training does not start
I'm running into following problem:
I've downloaded the horse2zebra dataset as descirbed and then wanted to train the model like that:
CUDA_VISIBLE_DEVICES=0 python main.py --dataset_dir=horse2zebra
Then i get following output:
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:4: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:
- https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
- https://github.com/tensorflow/addons
- https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:53: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:45: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
W0715 12:24:22.502655 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:45: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:47: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
W0715 12:24:22.502987 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/main.py:47: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-07-15 12:24:22.508503: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz 2021-07-15 12:24:22.508718: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a230300f40 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-07-15 12:24:22.508752: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-07-15 12:24:22.510533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-07-15 12:24:22.772124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.772938: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a2303012c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-07-15 12:24:22.772980: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2021-07-15 12:24:22.773161: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.773705: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:04.0 2021-07-15 12:24:22.774026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2021-07-15 12:24:22.775408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-07-15 12:24:22.776836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2021-07-15 12:24:22.777167: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2021-07-15 12:24:22.778532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2021-07-15 12:24:22.779210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2021-07-15 12:24:22.782099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-07-15 12:24:22.782220: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.782812: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.783323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-07-15 12:24:22.783399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2021-07-15 12:24:22.784766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-07-15 12:24:22.784805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-07-15 12:24:22.784816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-07-15 12:24:22.784944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.785715: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-07-15 12:24:22.786290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0) WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:44: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
W0715 12:24:22.787479 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:44: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/module.py:96: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
W0715 12:24:22.793503 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/module.py:96: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/module.py:101: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.
W0715 12:24:22.793844 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/module.py:101: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__
method instead.
W0715 12:24:22.796115 140356891568000 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__
method instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:15: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.
W0715 12:24:22.805560 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:15: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:15: calling RandomNormal.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0715 12:24:22.805814 140356891568000 deprecation.py:506] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:15: calling RandomNormal.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:19: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.
W0715 12:24:22.816351 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/ops.py:19: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:89: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
W0715 12:24:25.016935 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:89: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:92: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.
W0715 12:24:25.019891 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:92: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:115: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.
W0715 12:24:25.590578 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:115: The name tf.trainable_variables is deprecated. Please use tf.compat.v1.trainable_variables instead.
generatorA2B/g_e1_c/Conv/weights:0 generatorA2B/g_e1_bn/scale:0 generatorA2B/g_e1_bn/offset:0 generatorA2B/g_e2_c/Conv/weights:0 generatorA2B/g_e2_bn/scale:0 generatorA2B/g_e2_bn/offset:0 generatorA2B/g_e3_c/Conv/weights:0 generatorA2B/g_e3_bn/scale:0 generatorA2B/g_e3_bn/offset:0 generatorA2B/g_r1_c1/Conv/weights:0 generatorA2B/g_r1_bn1/scale:0 generatorA2B/g_r1_bn1/offset:0 generatorA2B/g_r1_c2/Conv/weights:0 generatorA2B/g_r1_bn2/scale:0 generatorA2B/g_r1_bn2/offset:0 generatorA2B/g_r2_c1/Conv/weights:0 generatorA2B/g_r2_bn1/scale:0 generatorA2B/g_r2_bn1/offset:0 generatorA2B/g_r2_c2/Conv/weights:0 generatorA2B/g_r2_bn2/scale:0 generatorA2B/g_r2_bn2/offset:0 generatorA2B/g_r3_c1/Conv/weights:0 generatorA2B/g_r3_bn1/scale:0 generatorA2B/g_r3_bn1/offset:0 generatorA2B/g_r3_c2/Conv/weights:0 generatorA2B/g_r3_bn2/scale:0 generatorA2B/g_r3_bn2/offset:0 generatorA2B/g_r4_c1/Conv/weights:0 generatorA2B/g_r4_bn1/scale:0 generatorA2B/g_r4_bn1/offset:0 generatorA2B/g_r4_c2/Conv/weights:0 generatorA2B/g_r4_bn2/scale:0 generatorA2B/g_r4_bn2/offset:0 generatorA2B/g_r5_c1/Conv/weights:0 generatorA2B/g_r5_bn1/scale:0 generatorA2B/g_r5_bn1/offset:0 generatorA2B/g_r5_c2/Conv/weights:0 generatorA2B/g_r5_bn2/scale:0 generatorA2B/g_r5_bn2/offset:0 generatorA2B/g_r6_c1/Conv/weights:0 generatorA2B/g_r6_bn1/scale:0 generatorA2B/g_r6_bn1/offset:0 generatorA2B/g_r6_c2/Conv/weights:0 generatorA2B/g_r6_bn2/scale:0 generatorA2B/g_r6_bn2/offset:0 generatorA2B/g_r7_c1/Conv/weights:0 generatorA2B/g_r7_bn1/scale:0 generatorA2B/g_r7_bn1/offset:0 generatorA2B/g_r7_c2/Conv/weights:0 generatorA2B/g_r7_bn2/scale:0 generatorA2B/g_r7_bn2/offset:0 generatorA2B/g_r8_c1/Conv/weights:0 generatorA2B/g_r8_bn1/scale:0 generatorA2B/g_r8_bn1/offset:0 generatorA2B/g_r8_c2/Conv/weights:0 generatorA2B/g_r8_bn2/scale:0 generatorA2B/g_r8_bn2/offset:0 generatorA2B/g_r9_c1/Conv/weights:0 generatorA2B/g_r9_bn1/scale:0 generatorA2B/g_r9_bn1/offset:0 generatorA2B/g_r9_c2/Conv/weights:0 generatorA2B/g_r9_bn2/scale:0 generatorA2B/g_r9_bn2/offset:0 generatorA2B/g_d1_dc/Conv2d_transpose/weights:0 generatorA2B/g_d1_bn/scale:0 generatorA2B/g_d1_bn/offset:0 generatorA2B/g_d2_dc/Conv2d_transpose/weights:0 generatorA2B/g_d2_bn/scale:0 generatorA2B/g_d2_bn/offset:0 generatorA2B/g_pred_c/Conv/weights:0 generatorB2A/g_e1_c/Conv/weights:0 generatorB2A/g_e1_bn/scale:0 generatorB2A/g_e1_bn/offset:0 generatorB2A/g_e2_c/Conv/weights:0 generatorB2A/g_e2_bn/scale:0 generatorB2A/g_e2_bn/offset:0 generatorB2A/g_e3_c/Conv/weights:0 generatorB2A/g_e3_bn/scale:0 generatorB2A/g_e3_bn/offset:0 generatorB2A/g_r1_c1/Conv/weights:0 generatorB2A/g_r1_bn1/scale:0 generatorB2A/g_r1_bn1/offset:0 generatorB2A/g_r1_c2/Conv/weights:0 generatorB2A/g_r1_bn2/scale:0 generatorB2A/g_r1_bn2/offset:0 generatorB2A/g_r2_c1/Conv/weights:0 generatorB2A/g_r2_bn1/scale:0 generatorB2A/g_r2_bn1/offset:0 generatorB2A/g_r2_c2/Conv/weights:0 generatorB2A/g_r2_bn2/scale:0 generatorB2A/g_r2_bn2/offset:0 generatorB2A/g_r3_c1/Conv/weights:0 generatorB2A/g_r3_bn1/scale:0 generatorB2A/g_r3_bn1/offset:0 generatorB2A/g_r3_c2/Conv/weights:0 generatorB2A/g_r3_bn2/scale:0 generatorB2A/g_r3_bn2/offset:0 generatorB2A/g_r4_c1/Conv/weights:0 generatorB2A/g_r4_bn1/scale:0 generatorB2A/g_r4_bn1/offset:0 generatorB2A/g_r4_c2/Conv/weights:0 generatorB2A/g_r4_bn2/scale:0 generatorB2A/g_r4_bn2/offset:0 generatorB2A/g_r5_c1/Conv/weights:0 generatorB2A/g_r5_bn1/scale:0 generatorB2A/g_r5_bn1/offset:0 generatorB2A/g_r5_c2/Conv/weights:0 generatorB2A/g_r5_bn2/scale:0 generatorB2A/g_r5_bn2/offset:0 generatorB2A/g_r6_c1/Conv/weights:0 generatorB2A/g_r6_bn1/scale:0 generatorB2A/g_r6_bn1/offset:0 generatorB2A/g_r6_c2/Conv/weights:0 generatorB2A/g_r6_bn2/scale:0 generatorB2A/g_r6_bn2/offset:0 generatorB2A/g_r7_c1/Conv/weights:0 generatorB2A/g_r7_bn1/scale:0 generatorB2A/g_r7_bn1/offset:0 generatorB2A/g_r7_c2/Conv/weights:0 generatorB2A/g_r7_bn2/scale:0 generatorB2A/g_r7_bn2/offset:0 generatorB2A/g_r8_c1/Conv/weights:0 generatorB2A/g_r8_bn1/scale:0 generatorB2A/g_r8_bn1/offset:0 generatorB2A/g_r8_c2/Conv/weights:0 generatorB2A/g_r8_bn2/scale:0 generatorB2A/g_r8_bn2/offset:0 generatorB2A/g_r9_c1/Conv/weights:0 generatorB2A/g_r9_bn1/scale:0 generatorB2A/g_r9_bn1/offset:0 generatorB2A/g_r9_c2/Conv/weights:0 generatorB2A/g_r9_bn2/scale:0 generatorB2A/g_r9_bn2/offset:0 generatorB2A/g_d1_dc/Conv2d_transpose/weights:0 generatorB2A/g_d1_bn/scale:0 generatorB2A/g_d1_bn/offset:0 generatorB2A/g_d2_dc/Conv2d_transpose/weights:0 generatorB2A/g_d2_bn/scale:0 generatorB2A/g_d2_bn/offset:0 generatorB2A/g_pred_c/Conv/weights:0 discriminatorB/d_h0_conv/Conv/weights:0 discriminatorB/d_h1_conv/Conv/weights:0 discriminatorB/d_bn1/scale:0 discriminatorB/d_bn1/offset:0 discriminatorB/d_h2_conv/Conv/weights:0 discriminatorB/d_bn2/scale:0 discriminatorB/d_bn2/offset:0 discriminatorB/d_h3_conv/Conv/weights:0 discriminatorB/d_bn3/scale:0 discriminatorB/d_bn3/offset:0 discriminatorB/d_h3_pred/Conv/weights:0 discriminatorA/d_h0_conv/Conv/weights:0 discriminatorA/d_h1_conv/Conv/weights:0 discriminatorA/d_bn1/scale:0 discriminatorA/d_bn1/offset:0 discriminatorA/d_h2_conv/Conv/weights:0 discriminatorA/d_bn2/scale:0 discriminatorA/d_bn2/offset:0 discriminatorA/d_h3_conv/Conv/weights:0 discriminatorA/d_bn3/scale:0 discriminatorA/d_bn3/offset:0 discriminatorA/d_h3_pred/Conv/weights:0 WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:40: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
W0715 12:24:25.591871 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:40: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:123: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
W0715 12:24:25.703997 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:123: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/math_grad.py:1375: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0715 12:24:25.767748 140356891568000 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/ops/math_grad.py:1375: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:128: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
W0715 12:24:36.170656 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:128: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
WARNING:tensorflow:From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:130: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
W0715 12:24:38.435485 140356891568000 module_wrapper.py:139] From /content/drive/MyDrive/MA/CycleGan/CycleGAN-tensorflow-2/CycleGAN-tensorflow-master/model.py:130: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.
After a couple of seconds the program termiantes (wihtout an error or anything), and so the training never starts.
If i check tensoroard it only shows me a visualization of the network.
Any idea what is going wrong?
Thanks a lot