DIGITS icon indicating copy to clipboard operation
DIGITS copied to clipboard

Creating tensorflow model fails

Open pq53ui opened this issue 5 years ago • 0 comments

Python: 2.7.12 Tensorflow: 1.14.0 caffe: 1.0.0 OS: Ubuntu 16.04.4 LTS

I am trying to create a new tensorflow image classification model using mnist dataset and standard network LeNet provided by DIGITS. Using caffe option for LeNet, there are no problems with model creation and training. But if I use tensorflow option I get the following error:

Traceback (most recent call last):
  File "/root/digits/digits/scheduler.py", line 512, in run_task
    task.run(resources)
  File "/root/digits/digits/task.py", line 227, in run
    for line in utils.nonblocking_readlines(self.p.stdout):
  File "/root/digits/digits/utils/__init__.py", line 74, in nonblocking_readlines
    yield buf[:r].decode(enc) + '\n'
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdc in position 61: ordinal not in range(128)

I am not using any special characters... anywhere. Any ideas where the problem might be? The full debugger trace from DIGITS:

2019-07-19 11:04:37 [20190719-110436-f4f0] [INFO ] Train Tensorflow Model task started.

2019-07-19 11:04:37 [20190719-110436-f4f0] [INFO ] Task subprocess args: "/usr/bin/python2 /root/digits/digits/tools/tensorflow/main.py --network=network.py --epoch=30 --networkDirectory=/root/digits/digits/jobs/20190719-110436-f4f0 --save=/root/digits/digits/jobs/20190719-110436-f4f0 --snapshotPrefix=snapshot --snapshotInterval=1.0 --lr_base_rate=0.01 --lr_policy=step --mean=/root/digits/digits/jobs/20190715-133340-32e7/mean.binaryproto --labels_list=/root/digits/digits/jobs/20190715-133340-32e7/labels.txt --train_db=/root/digits/digits/jobs/20190715-133340-32e7/train_db --validation_db=/root/digits/digits/jobs/20190715-133340-32e7/val_db --lr_gamma=0.1 --lr_stepvalues=33.0 --shuffle=1 --subtractMean=image --optimization=sgd --validation_interval=1 --log_runtime_stats_per_step=0 --augFlip=none --augHSVh=0 --augHSVs=0 --augHSVv=0"

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: WARNING: Logging before flag parsing goes to stderr.

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: W0719 11:04:39.664391 140228124428032 deprecation_wrapper.py:119] From /root/digits/digits/tools/tensorflow/main.py:743: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead. 2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.666380 140228124428032 main.py:417] Train batch size is 16 and validation batch size is 16

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.666531 140228124428032 main.py:421] Training epochs to be completed for each validation : 1

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.667139 140228124428032 main.py:425] Training epochs to be completed before taking a snapshot : 1.0

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.667377 140228124428032 main.py:429] Model weights will be saved as snapshot_<EPOCH>_Model.ckpt

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.667623 140228124428032 main.py:442] Loading mean tensor from /root/digits/digits/jobs/20190715-133340-32e7/mean.binaryproto file

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.669307 140228124428032 main.py:448] Loading label definitions from /root/digits/digits/jobs/20190715-133340-32e7/labels.txt file

2019-07-19 11:04:39 [20190719-110436-f4f0] [WARNING] Train Tensorflow Model unrecognized output: I0719 11:04:39.669481 140228124428032 main.py:454] Found 10 classes

2019-07-19 11:04:39 [20190719-110436-f4f0] [ERROR] UnicodeDecodeError: 'ascii' codec can't decode byte 0xdc in position 61: ordinal not in range(128)

pq53ui avatar Jul 19 '19 11:07 pq53ui