Daniel Galvez comments

Results 85 comments of


                                            Daniel Galvez

Batched streaming RNN-T and TDT: new buffered inference + support cache-aware

@titu1994 if you could rereview at your convenience, that would be appreciated.

Fix #8891 by supported GPU-side batched CTC Greedy Decoding

To verify that latency does indeed fall, I ran the above prefixed by `nsys profile --env-var=NSYS_NVTX_PROFILER_REGISTER_ONLY=0 -t nvtx -c nvtx -p GALVEZ_START` with batch_inference=false and batch_inference=true. I added an nvtx...

Fix #8891 by supported GPU-side batched CTC Greedy Decoding

Will need to get back to you about what it takes to make this the default algo. But the main concern is that many existing ".nemo" files will have the...

Fix #8891 by supported GPU-side batched CTC Greedy Decoding

@titu1994 done. To clarify, I went with the warn-once path rather than making "greedy" pointed to the new implementation.

Consider making the decoding classes subclasses of torch.nn.Module and registering them via add_module()

My overall conclusion is that I have encountered an edge case, and the right approach is just to recreate the appropriate state tensors anytime that the device of the input...

Consider making the decoding classes subclasses of torch.nn.Module and registering them via add_module()

Closing this. The better way is lazy initialization given how nemo currently is.

Dgalvez/cuda graphs greedy rnnt inference squash

This PR is ready for review. @artbataev @titu1994 would you be willing?

Dgalvez/cuda graphs greedy rnnt inference squash

@artbataev Let me know if you think this is good to go.

Dgalvez/cuda graphs greedy rnnt inference squash

@artbataev thank you for the initial suggestion. It works when the decoder has not been run yet. However, it doesn't work if the decoder has already been run. You can...

Dgalvez/cuda graphs greedy rnnt inference squash

jenkins