tf.train.import_meta_graph throws a ValueError

[Aravind-Suresh]

Description

I downloaded the pre-trained model here ( provided in README ) and tried loading the model. But I was not able to import the graph definitions from the meta file.

Environment info

Operating System: Ubuntu 14.04.3 LTS Installed version of CUDA: 8.0.0 pip package: https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-0.12.1-cp34-cp34m-linux_x86_64.whl Tensorflow version ( import tensorflow as tf; print(tf.VERSION); ): 0.12.1

Code to reproduce

import os
import sys
import time
import json
import argparse

import numpy as np
import tensorflow as tf

# Model downloaded to models folder
saver = tf.train.import_meta_graph("models/params_cifar.ckpt.meta")

# Throws error
# ValueError: Shape must be at most rank 1 but is rank 2 for 'model/conv2d_0/l2_normalize/Sum' (op: 'Sum') with input shapes: [2,3,4,160], [1,3].

Trace

$ python3 snips/load_model.py 
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/common_shapes.py", line 670, in _call_cpp_shape_fn_impl
    status)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shape must be at most rank 1 but is rank 2 for 'model/conv2d_0/l2_normalize/Sum' (op: 'Sum') with input shapes: [2,3,4,160], [1,3].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "snips/load_model.py", line 11, in <module>
    saver = tf.train.import_meta_graph("models/params_cifar.ckpt.meta")
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/saver.py", line 1526, in import_meta_graph
    **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/meta_graph.py", line 502, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/importer.py", line 380, in import_graph_def
    ops.set_shapes_for_outputs(op)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 1617, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 1568, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/common_shapes.py", line 675, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Shape must be at most rank 1 but is rank 2 for 'model/conv2d_0/l2_normalize/Sum' (op: 'Sum') with input shapes: [2,3,4,160], [1,3].

[alexggmatthews]

@Aravind-Suresh I get a similar problem with python 3.5.2 and tf 0.11.0 so the problem exists with a couple of different tf versions at least. Did you make any progress fixing this?

[Aravind-Suresh]

@alexggmatthews I tried different combinations of { tf versions, py versions }. tf.import_meta_graph works for tf 0.10 and py 2.7/3.4.

I guess the version of tensorflow is the bottleneck. Using v0.10 resolves this issue. But still I am having difficulties restoring the checkpoint because of CUDA version conflicts.

Tensorflow v0.10 works best with CUDA 7.5 and cuDNN v5. But later versions of tensorflow require CUDA 8.0 and cuDNN v5.1. Reinstalling a bunch of tools will fix the issue I guess.

Let me know if you have a workaround for this.