bert
bert copied to clipboard
the set of log_step_count_steps doesn't work in tf.contrib.tpu.RunConfig
I wanna control the info message "examples/sec" appearance. Cause I use the google colab to train the model, the more info appear, the browser would be stuck.
So I set the log_step_count_steps param.
But it's wired I change the code as below, but it still output:'_log_step_count_steps': None
.
run_config = tf.contrib.tpu.RunConfig(
cluster=tpu_cluster_resolver,
master=FLAGS.master,
model_dir=FLAGS.output_dir,
save_checkpoints_steps=FLAGS.save_checkpoints_steps,
log_step_count_steps = FLAGS.log_step_count_steps,
tpu_config=tf.contrib.tpu.TPUConfig(
iterations_per_loop=FLAGS.iterations_per_loop,
num_shards=FLAGS.num_tpu_cores,
per_host_input_for_training=is_per_host))
Can anyone help me? Appreciate for any help.
The tensorflow version is 1.12.0.
I'm having this issue in TF 1.14
Even when manually adding the hook to the training function it just vanishes. Some TPU magic doing away with it?
import tensorflow_estimator.python.estimator.tpu.tpu_estimator as tpe
epshook = tpe.ExamplesPerSecondHook(args._batch_size,output_dir=args.result_dir,every_n_steps=100)
train_estimator.train(input_fn=train_input_fn, hooks=[epshook])
any solution to this problem?