Patrick Nguyen

Did you look at the images? On Sun, Jul 7, 2019 at 6:12 PM boji123 wrote: > hello! > I opened the event output using tensorboard, but I couldn't find...

Right, it seems that the GPU utilization is pretty low. Can you try some more tensorflow profiling to see where the time is spent?

OK, I'm not an expert in reading traces. I don't really see a smoking gun. Can you follow the [instructions](https://cloud.google.com/blog/products/ai-machine-learning/profiling-the-model-training-process-for-tensorflow-on-cloud-ml-engine) to profile, especially trying "advise" and the profiler-ui to see...

The protobuf version is set [here](https://github.com/tensorflow/lingvo/blob/master/lingvo/repo.bzl#L245). You can check the history of that file to see the sha256. If you use an older version, you are on your own in...

It seems that bazel changed the output. Feel free to send a PR to update the README and `run_distributed`. On Tue, Sep 17, 2019 at 11:41 AM Ondrej Platek wrote:...

Could you check: ``` python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())' ``` and see if that file (`tensorflow/core/platform/threadpool.h`) lives under that directory? It's best to build the whole thing: ```...

@kazuki-irie @rprabhavalkar any comment?

You can try to switch to `bfloat16`. That's half the size. During training, you may want to keep training internally in `float32` for more stability.

Are you running on py3 by any chance? On Mon, Aug 19, 2019 at 2:41 PM alessiaatunimi wrote: > Hi, > I'm trying to download preprocess the dataset for the...

Nothing springs to mind. Can you print `type(l)`?