yinjunbo

Besides, when I trained StarNet on KITTI with `bazel-bin/lingvo/trainer`, it failed with ``` ValueError: Failed to create a one-shot iterator for a dataset. `Dataset.make_one_shot_iterator()` does not support datasets that capture...

The info for p.groundtruth_database, ``` Loading groundtruth database at { allow_implicit_capture: None cls: dtype: file_pattern: "kitti_train_object_cls.tfrecord-00000-of-00100" file_pattern_prefix: "/home/junbo/datasets/KITTI/kitti_object/starnet-tfr/" file_type: "" fprop_dtype: None inference_driver_name: None is_inference: None name: "datasource" params_init: {...

@vrv, I have specified a root path containing to all of the input files, and the output file `train.tfr-00000-of-00001` seems to include all the 798 segments with a size of...

@vrv Thanks for your advice for training locally. As for the dataset format, in my case, I run it with ``` python generate_waymo_tf.py --input_file_pattern=/path_to_waymo_data/segment-*_with_camera_labels.tfrecord --output_filebase=/path_to_preprocessed_data/train.tfr@1000 ``` Then, it just output...

@vrv Following your advice of adding `num_shards=1000`, I can now obtain the correct file number. However, it will cost a long time to prcoess them with a local machine and...

@vrv, Thanks a lot for providing the preprocessed data! BTW, I find the test TF.Example is just 100MB, which is much little than train (~5G). Is it all right?

I'm now try to train with TPU cluster, but there is an issue: ``` Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line...

@vrv ,Could you run you code on GKE successfully? What's your tensorflow version for running on TPU? I find It's not a issue caused by `ComputationShape`. In fact, the `topology`...

@vrv , I can train locally with 8*32G GPUs, but the bach size should be 1 that leads to a very long training time. I have changed ComputationShape() to length...

However, I cannot compile the trainer `bazel build -c opt //lingvo:trainer` with tf-nightly-2.3. ``` ERROR: /home/junbo/repository/starnet/lingvo/lingvo/core/ops/BUILD:474:1: C++ compilation of rule '//lingvo/core/ops:hyps_proto' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc...