lingvo

lingvo copied to clipboard

How to run microbatches on different gpus?

3

Hi, I've been experimenting with GPipe and was wondering if it is possible to run different micro-batches on different GPUS? For example if there are 16 micro-batches, is it possible...

adis98

@bignamehyp Hi, when I try to run the Gpipe example one_billion_wds using the given command: trainer --run_locally=gpu --mode=sync --model=lm.one_billion_wds.OneBWdsGPipeTransformerWPM --logdir=/tmp/lm/log --logtostderr --worker_split_size=4 --worker_gpus=4 There is an error : Traceback (most...

xsppp

Questions about ASR frontend, specifically Librispeech, feature normalization

1

Hi, I'm checking your ASR frontend, specifically the Librispeech audio feature extraction, and have some questions. References: [Librispeech params](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tasks/asr/params/librispeech.py), [ASR encoder](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tasks/asr/encoder.py), [ASR model](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tasks/asr/model.py), [create ASR features](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tools/create_asr_features.py), [`ExtractLogMelFeatures`](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tools/audio_lib.py#L72), [`MelAsrFrontend`](https://github.com/tensorflow/lingvo/blob/d4487348c08fe6e9757308fab8c3c7c609f0593d/lingvo/tasks/asr/frontend.py#L115) I don't...

albertz

there is no pip package on windows?

1

when I run "pip3 install lingvo", Receive prompt : "ERROR: Could not find a version that satisfies the requirement lingvo (from versions: none) ERROR: No matching distribution found for lingvo"...

Joseph-sun

Training time

23

i have training ASR tasks by 4 GPU sync mode and async mode, but the training was so slow; that is training log: ``` INFO:tensorflow:time:6.841992 INFO:tensorflow:2019.03.30-21:26:33 step: 24 fraction_of_correct_next_step_preds:0 fraction_of_correct_next_step_preds/logits:0...

dingevin

Any plans or timelines to release gshard code?

3

Great work on gshard, curious if there are any plans to open-source it and if so, timeline?

ngoyal2707

GradDrop renormalization

2

@jngiam Is this correct? I would expect the norms in the quotient to be the other way around. https://github.com/tensorflow/lingvo/blob/cbceb3add8932554cb7a0aaa1a823d58fbd2b59e/lingvo/core/graddrop.py#L184-L185 Thanks

thomsonn

Run resnet-101 with gpipe?

1

Do anyone have a example run resnet-50/101 with Gpipe from the Gpipe paper?

gaokai0810

Does GShard support GPUs？

1

Hi, I can see that GShard can run large models on TPUs, does it also support GPUs now ?

aoyulong

Thank you very much for open-sourcing GShard! I'm currently using MoE from Mesh Tensorflow. The design of MoE used in MTF is equivalent to that of GShard iiuc. According to...

AranKomat