nlp-paper

nlp-paper copied to clipboard

Performance issue in /paper-code/tensorflow_src/tools (by P3)

1

Hello! I've found a performance issue in preprocess_tfrecord.py: `dataset.batch(batch_size, drop_remainder=drop_remainder)`[(line 179)](https://github.com/DengBoCong/nlp-paper/blob/60dfa0688d9e95d5dcce9586efcb045d8d47c7ac/paper-code/tensorflow_src/tools/preprocess_tfrecord.py#L179) should be called before `dataset.map(map_func=_parse_dataset_item, num_parallel_calls=mt.cpu_count())`[(line 173)](https://github.com/DengBoCong/nlp-paper/blob/60dfa0688d9e95d5dcce9586efcb045d8d47c7ac/paper-code/tensorflow_src/tools/preprocess_tfrecord.py#L173), which could make your program more efficient. Here is [the tensorflow document](https://tensorflow.google.cn/guide/data_performance?hl=zh_cn#vectorized_mapping)...

DLPerf