benchmarks

benchmarks copied to clipboard

Horovod converges slow for resnet

21

When using real ImageNet datasets instead synthetic ones, we found horovod converges much slower than replicated with NCCL **only on ResNet**. We are aware of the fix #190 by @alsrgv...

jiajinyu

The accuracy of training resnet50 on the Imagenet2012 dataset is lower than 75%

10

Base on the research of 'Kaiming He https://github.com/KaimingHe/deep-residual-networks', the accuracy of training resnet50 on the Imagenet2012 dataset should be around 75%. The alexnet model should be around 58% described in...

Sampson1107

Weight decay of biases

1

Hi, is it intended to perform weight decay on biases as it seems to be the case of the code and it seems that performing weight decay on biases will...

YingHaoTan

How to run the benchmark in the distributed mode?

20

Hi, I followed the instructions from the [performance page]{https://www.tensorflow.org/performance/performance_models}, and run on two EC2 p2.8xlarge instances, using the same benchmark hash (Benchmark GitHub hash: 9165a70). ``` # Run the following...

yupeng9

OSError on 128 GPUs for distributed_replicated on AWS P3

8

Hi, I'm trying to run a distributed replicated benchmark with 128 V100s and I'm getting a OSError. Some more details: - Using AWS P3 instances (16 of them) - Batch...

richardliaw

How to split dataset by multi workers?

2

HI, if i have 5 parts input dataset in hdfs, then if i use 5 workers to train 2 epochs i think worker 0 read part-0 2epochs, worker 1 read...

anpark

Distributed Tensorflow fails with "Waiting for model to be ready. Ready_for_local_init_op: Variables not initialized" log msg

7

Tried running distributed Tensorflow between 3 machines, with 1 as PS and the other 2 as workers. The PS and one of the workers start good as expected. However the...

antoajayraj

tensorflow.python.framework.errors_impl.NotFoundError:

2

I am able to run this benchmark using P100 with python 2.7, CUDA9, cudnn 7, and TF 1.6. I am trying to run the same with K20 using the same...

abidmalikwaterloo

Using batch_size * num_gpus as batch_size in exponential_decay calculation?

5

Hi guys, I've been reading benchmark_cnn.py and trying to figure out what batch_size really is in a multi-GPU setup when using it to calculate num_batches_per_epoch and decay_steps. num_batches_per_epoch = (float(num_examples_per_epoch)...

h8907283

Keras+Tensorflow Benchmark on Synthetic LSTM Dataset

1

Hi, I am running the `lstm_benchmark.py` test on CPU and multi GPU device(Amazon EC2) and I am not getting scaling as expected. Below are the pieces of information: **Instance:** P3.8xLarge(Amazon...

karan6181