bert icon indicating copy to clipboard operation
bert copied to clipboard

the set of log_step_count_steps doesn't work in tf.contrib.tpu.RunConfig

Open jimmy-walker opened this issue 5 years ago • 2 comments

I wanna control the info message "examples/sec" appearance. Cause I use the google colab to train the model, the more info appear, the browser would be stuck. So I set the log_step_count_steps param. But it's wired I change the code as below, but it still output:'_log_step_count_steps': None.

    run_config = tf.contrib.tpu.RunConfig(
        cluster=tpu_cluster_resolver,
        master=FLAGS.master,
        model_dir=FLAGS.output_dir,
        save_checkpoints_steps=FLAGS.save_checkpoints_steps,
        log_step_count_steps = FLAGS.log_step_count_steps,
        tpu_config=tf.contrib.tpu.TPUConfig(
            iterations_per_loop=FLAGS.iterations_per_loop,
            num_shards=FLAGS.num_tpu_cores,
            per_host_input_for_training=is_per_host))

Can anyone help me? Appreciate for any help.

The tensorflow version is 1.12.0.

jimmy-walker avatar Jul 26 '19 06:07 jimmy-walker

I'm having this issue in TF 1.14

Even when manually adding the hook to the training function it just vanishes. Some TPU magic doing away with it?

import tensorflow_estimator.python.estimator.tpu.tpu_estimator as tpe

epshook = tpe.ExamplesPerSecondHook(args._batch_size,output_dir=args.result_dir,every_n_steps=100)

train_estimator.train(input_fn=train_input_fn, hooks=[epshook])

AlLongley avatar Oct 04 '19 14:10 AlLongley

any solution to this problem?

crystina-z avatar Apr 01 '20 21:04 crystina-z