Vadim Kantorov

`export CFLAGS=-std=c++14` didn't help

Compiler calls are still like `gcc -pthread -B /miniconda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++14 -fPIC -I/tmp/pip-req-build-8t7lyg7q/third_party/kenlm -I/tmp/pip-req-build-8t7lyg7q/third_party/openfst-1.6.7/src/include -I/tmp/pip-req-build-8t7lyg7q/third_party/ThreadPool -I/tmp/pip-req-build-8t7lyg7q/third_party/boost_1_67_0 -I/tmp/pip-req-build-8t7lyg7q/third_party/utf8 -I/miniconda/lib/python3.7/site-packages/torch/include -I/miniconda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/miniconda/lib/python3.7/site-packages/torch/include/TH -I/miniconda/lib/python3.7/site-packages/torch/include/THC -I/miniconda/lib/python3.7/site-packages/torch/include -I/miniconda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include...

Probably https://github.com/parlance/ctcdecode/blob/master/build.py#L44 is problematic and needs to be updated to c++14

Confirmed, this is the cause. `sed -i 's/11/14/g' build.py` fixes the problem

> A previous [repot](https://github.com/pytorch/pytorch/issues/17798) indicated that native torch CTC loss might not reproducible on GPU under certain conditions. I found that [ESPnet](https://github.com/espnet/espnet/blob/master/espnet/nets/pytorch_backend/ctc.py#L68) also encounter the same issue and they patched...

Also, float16 has max value of 65536 and quite inaccurate in the higher range, so using it for length calculation (if at all) should be done with extreme caution

Also for float16 is badly supported on CPU (https://github.com/pytorch/pytorch/issues/80483#issuecomment-1169404072), it should probably be changed to default int64 or float32 that are supported on most devices. Probably positional encoding computation should...

@popcornell Do you mean model accuracy was much worse? or computation speed? did you get any nans/infs while training?

Here's Nemo's impl of positional encoding (also includes caching): https://github.com/NVIDIA/NeMo/blob/633674898132b315b4f2eaf3761de391d3d0c1f4/nemo/collections/asr/parts/submodules/multi_head_attention.py#L231-L244

My usecase from #12232 (dupe of this one): "When ssh hangs in WSLv1 (usually because of network issues or after laptop sleep / hibernate), somehow it doesn't react to any...