How to Evaluate Model?

[WilliamCarlos]

I trained a model using python trainer.py --dataset CIFAR10 and evaluated it via python evaler.py --dataset CIFAR10 --checkpoint train_dir/default-CIFAR10_lr_0.0001_update_G5_D1-20171209-215809/model-99001 (with 99001 being chosen somewhat arbitrarily).

The resulting output is:

[2017-12-09 22:52:02,675] dataset: CIFAR10
[2017-12-09 22:52:05,980] Loaded from checkpoint!
[2017-12-09 22:52:05,981] Start 1-epoch Inference and Evaluation
[2017-12-09 22:52:05,981] # of examples = 10000
[2017-12-09 22:52:05,981] max_steps = 1
[2017-12-09 22:52:07,615]  [val   step    0] (1.631 sec/batch, 39.250 instances/sec)

How do I interpret this result? I tried specifying an output_file but the results were unclear (binary file?). Thanks in advance!

[shaohua0116]

The result would be saved as an hdf5 file. For example, you can save the result using this command: python evaler.py --dataset CIFAR10 --checkpoint train_dir/default-CIFAR10_lr_0.0001_update_G5_D1-20171209-215809/model-99001 --output_file output.hdf5. This file can be opened with the h5py package.

[WilliamCarlos]

Thanks for the prompt response! Examining output.hdf5 it appears the only entry is "images"? Is it possible to get classification accuracies?

Also, I used the following code to break the 64x32x32x3 block of images into 64 32x32 RGB images:

import numpy as np
import h5py
from PIL import Image
import pdb

hdf = h5py.File('output_5000.hdf5','r')

# ls = list(hdf.keys())
# print(ls)

data = hdf.get('image')

# 64 x 32 x 32 x 3 block (64 RGB images)
data_array = np.array(data)

for i in range(64):
    image = data_array[i,:,:,:]
    image = Image.fromarray(image, 'RGB')
    image.save('images/{0}.png'.format(i))

But I'm getting some strange results: (from model 5000) (from model 1001) (from model 99001)

Do you have any thoughts on why the generated images look like this?

[WilliamCarlos]

Running visualize_trainering.py:

[shaohua0116]

Thanks for providing detailed descriptions of the issue. It looks like your model was not trained well. As DCGAN model is sensitive to the weight initialization, I would recommend you to try to run several more jobs. Also, the checkpoint is provided here. You can download the checkpoints to verify if your visualization codes are correct.

[WilliamCarlos]

Hmm.. I'm guessing my visualization code is not correct because I'm getting the same result even using the checkpoint model. Do you happen to have any visualization code lying around? If not, I'll play around with it and see if I can get it working. Thanks!

[maryam089]

Can You explain your statement ? @WilliamCarlos "DCGAN model is sensitive to the weight initialization "