k2
k2 copied to clipboard
k2-fsa
fail k2.compose(H, LG, inner_labels="tokens")
when i run local/test.py, fail in line 136 --->HLG = k2.compose(H, LG, inner_labels="tokens")
error:
/var/www/k2/csrc/array_ops.cu:279:void k2::RowSplitsToRowIds(const k2::Array1
[ Stack-Trace: ]
/opt/conda/lib/python3.10/site-packages/k2/lib64/libk2_log.so(k2::internal::GetStackTrace()+0x34) [0x7f0e202a19b4]
/opt/conda/lib/python3.10/site-packages/k2/lib64/libk2context.so(k2::internal::Logger::~Logger()+0x2a) [0x7f0e2083cd4a]
/opt/conda/lib/python3.10/site-packages/k2/lib64/libk2context.so(k2::RowSplitsToRowIds(k2::Array1
Traceback (most recent call last):
File "/home/liweiyong/project/icefall/egs/commonvoice/ASR/./local/compile_hlg.py", line 168, in
gdb --args python /path/to/your/code.py
(You can use `gdb` to debug the code. Please consider compiling
a debug version of k2.).
If you are unable to fix it, please open an issue at:
https://github.com/k2-fsa/k2/issues/new
Can you clarify the size of your H and LG? I suspect it is an overflow of int32.
2024-05-10 03:23:26,208 INFO [compile_hlg.py:133] H shape before Composing HLG: (501, None) 2024-05-10 03:23:26,208 INFO [compile_hlg.py:134] LG shape before Composing HLG: (11745505, None)
Similar to #1010. Any way to solve this?
That's true, Smaller LG Compose succeeded （H.shape: 501, LG.shape: 11745505）, the larger LG Compose fail （H.shape: 501, LG.shape: 51395348）. the larger LG there is no other good way to Compose. how much is the upper limit of Compose
If this is a graph-building exercise of some kind it may be better to use the kaldifst project https://k2-fsa.github.io/kaldifst/ to do the graph-building, then if needed it can be converted to the k2 format.
If this is a graph-building exercise of some kind it may be better to use the kaldifst project https://k2-fsa.github.io/kaldifst/ to do the graph-building, then if needed it can be converted to the k2 format.
I've tried KaldiFST following the PR here, but the result just did not change.
What do you mean "just did not change"?
I got the same error as above, after the optional OpenFST procedures.
You mean you can not compile HLG even with openfst?
By using openfst, you should convert H and LG from k2 to openfst, then compile HLG using openfst, convert the HLG back to k2 at last.
What do you mean "just did not change"?
I got the same error as above, after the optional OpenFST procedures.
Could you show the error logs after switching to kaldifst?
What do you mean "just did not change"?
I got the same error as above, after the optional OpenFST procedures.
Could you show the error logs after switching to kaldifst?
Sorry that I didn't save the logs. I followed the LibriSpeech script in egs/librispeech/ASR/local/compile_hlg_using_openfst.py.
It started running slowly with openfst for several hours, and the same error as mentioned in this issue happened again when composing HLG at Line-156.
My lm.arpa has 80 million grams with order 1 to 3.
openfst
k2 to openfst use "from kaldifst.utils import k2_to_openfst ---> H_fst = k2_to_openfst(H)". compose use "HLG_fst = kaldifst.compose(H_fst, LG_fst)"， it success. how to convert openfst to k2
openfst
k2 to openfst use "from kaldifst.utils import k2_to_openfst ---> H_fst = k2_to_openfst(H)". compose use "HLG_fst = kaldifst.compose(H_fst, LG_fst)"， it success. how to convert openfst to k2
Check files of https://github.com/k2-fsa/icefall/pull/606. The conversion between k2 and fst is successful, but I guess the problem here is that k2.compose is still used after converting fst to k2.