Decoder processes terminating prematurely

[victoryhb]

Hi Silva, Thanks a lot for the great work on the docker image which I have been experimenting for some time now.

The problem I am experiencing is that the worker process tends to terminate mysteriously after sustained usage for some time (say 1000 calls to the ASR server from another process from a web server) and the Python server usually returns the following error message:

Received error from server (status 9)
Error message: No decoder available, try again later

The last few lines from the worker.log file give the following:

2016-08-31 08:01:46 -   DEBUG:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Checking that decoder hasn't been silent for more than 10 seconds
2016-08-31 08:01:47 -   DEBUG:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Checking that decoder hasn't been silent for more than 10 seconds
2016-08-31 08:01:48 - WARNING:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: More than 10 seconds from last decoder hypothesis update, cancelling
2016-08-31 08:01:48 -    INFO:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Master disconnected before decoder reached EOS?
2016-08-31 08:01:48 -    INFO:   decoder2: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Sending EOS to pipeline in order to cancel processing
2016-08-31 08:01:48 -    INFO:   decoder2: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Cancelled pipeline
2016-08-31 08:01:48 -    INFO:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Waiting for EOS from decoder
ERROR (CopyRowsFromVec():kaldi-matrix.cc:864) Wrong sized arguments
2016-08-31 08:01:49 -   DEBUG:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Got message from server of type <class 'ws4py.messaging.BinaryMessage'>
2016-08-31 08:01:49 -    INFO:   __main__: ee8f8e2e-3678-43ea-a3b8-1789becf434c: Ignoring data, worker already in state 8
WARNING (RunNnetEvaluation():online-nnet2-decoding-threaded.cc:390) Caught exception: ERROR (CopyRowsFromVec():kaldi-matrix.cc:864) Wrong sized arguments

[stack trace: ]
kaldi::KaldiGetStackTrace()
kaldi::KaldiErrorMessage::~KaldiErrorMessage()
kaldi::MatrixBase<float>::CopyRowsFromVec(kaldi::VectorBase<float> const&)
kaldi::nnet2::NnetOnlineComputer::Flush(kaldi::CuMatrix<float>*)
kaldi::SingleUtteranceNnet2DecoderThreaded::RunNnetEvaluationInternal()
kaldi::SingleUtteranceNnet2DecoderThreaded::RunNnetEvaluation(void*)
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f78419670a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7840d7904d]
ERROR (WaitForAllThreads():online-nnet2-decoding-threaded.cc:426) Error encountered during decoding.  See above.
terminate called after throwing an instance of 'std::runtime_error'
  what():  ERROR (WaitForAllThreads():online-nnet2-decoding-threaded.cc:426) Error encountered during decoding.  See above.

[stack trace: ]
kaldi::KaldiGetStackTrace()
kaldi::KaldiErrorMessage::~KaldiErrorMessage()
kaldi::SingleUtteranceNnet2DecoderThreaded::WaitForAllThreads()
kaldi::SingleUtteranceNnet2DecoderThreaded::Wait()
/opt/gst-kaldi-nnet2-online/src/libgstkaldionline2.so(+0x53b6d) [0x7f783bb28b6d]
/opt/gst-kaldi-nnet2-online/src/libgstkaldionline2.so(+0x54ac6) [0x7f783bb29ac6]
/usr/lib/x86_64-linux-gnu/libgstreamer-1.0.so.0(+0x94ad6) [0x7f783db78ad6]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x712c8) [0x7f783f6ab2c8]
/lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x70935) [0x7f783f6aa935]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f78419670a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7840d7904d]

I have tried starting two workers, but the problems persist. What can be done to ensure the stable running of the docker server?

Victor

[jcsilva]

Hi,

please, could you also show me your master.log when this problem happened?

[victoryhb]

Hi Silva, thank you for your reply. Here's the (somewhat vague) output from master.log:

INFO 2016-08-31 08:01:12,267 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: OPEN
INFO 2016-08-31 08:01:12,267 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Request arguments: content-type=""
INFO 2016-08-31 08:01:12,268 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Using worker <__main__.DecoderSocketHandler object at 0x7eff01dfb050>
INFO 2016-08-31 08:01:12,506 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:12,762 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:13,022 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:13,282 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:13,542 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:18,380 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:18,383 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:19,422 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:19,426 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:19,482 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
WARNING 2016-08-31 08:01:20,430 405 CONNECT 163mx00.mxmail.netease.com:25 (59.122.126.28) 0.54ms
INFO 2016-08-31 08:01:21,444 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:21,448 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:21,451 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:22,495 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Sending event {u'status': 0, u'segment-start': 0.0, u'segment-length': 0.55, u'total-length': 0.55, u'result': ... to client
INFO 2016-08-31 08:01:22,530 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Sending event {u'status': 0, u'adaptation_state': {u'type': u'string+gzip+base64', u'id': u'0c4d8ad1-2d6b-4ea1-... to client
INFO 2016-08-31 08:01:22,531 Worker <__main__.WorkerSocketHandler object at 0x7eff01de8a50> leaving
INFO 2016-08-31 08:01:23,535 New worker available <__main__.WorkerSocketHandler object at 0x7eff01e69c90>
INFO 2016-08-31 08:01:27,535 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Handling on_connection_close()
INFO 2016-08-31 08:01:27,536 0c4d8ad1-2d6b-4ea1-824b-e5d01fd36400: Closing worker connection
INFO 2016-08-31 08:01:38,978 ee8f8e2e-3678-43ea-a3b8-1789becf434c: OPEN
INFO 2016-08-31 08:01:38,978 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Request arguments: content-type=""
INFO 2016-08-31 08:01:38,978 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Using worker <__main__.DecoderSocketHandler object at 0x7eff01e5f1d0>
INFO 2016-08-31 08:01:43,012 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:45,938 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:49,003 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:49,062 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Forwarding client message (<type 'str'>) of length 8000 to worker
INFO 2016-08-31 08:01:49,451 Worker <__main__.WorkerSocketHandler object at 0x7eff01e69c90> leaving
INFO 2016-08-31 08:01:49,840 ee8f8e2e-3678-43ea-a3b8-1789becf434c: Forwarding client message (<type 'str'>) of length 8000 to worker
ERROR 2016-08-31 08:01:49,840 Uncaught exception in /client/ws/speech
INFO 2016-08-31 08:01:49,841 ee8

Victor

[jcsilva]

Well,

I'll need some more time to debug this problem...

But, one question... you said that you used the system for some time (say 1000 calls to the ASR server from another process from a web server) without problems, right? All audio data you send to the worker is saved somewhere in your HD. Did you see if you are not facing some problems with disk full? It's just a guess...

[victoryhb]

I haven't found disk space to be of a problem since the audio files I send to the ASR server are relatively small (10k-800k) and the disk still has hundreds of gigabytes available. One thing that I observe is that when I start two workers, they tend to terminate one after the other, with an indefinite gap between their termination. One workaround I am thinking of is to start a cron job to check if one of the worker processes has terminated, and if yes then restart it... Got to be a better way out :)

[jcsilva]

Hi,

this weekend I tried, but I was not able to reproduce your problem. So, I uploaded a "new" docker image that I would ask you to test and confirm if this issue exists with it.

This docker image is jcsilva:docker-kaldi-gstreamer-server:tedlium. You don't need to configure anything to make it work. It already has an acoustic and language model inside it. So, please, just run docker run -p 9999:80 jcsilva:docker-kaldi-gstreamer-server:tedlium and then send your audio to port 9999. It should work without problems.

If you have any problem with this image, please, detail all your steps and I will reproduce the problem here and help you ... it's gonna be easier for me.

Thank you, Eduardo

[victoryhb]

Hi Eduardo, Thanks a lot for your very kind help and I am definitely gonna try your new image out and get back to you. Please bear with me if it takes some time as I am in the middle of an urgent project lately. :-)

P.S. I wonder if there will be further enhancements to the model in the image in the future? For example, can the confidence score of each word be provided in the returned results?

Thank you, Victor

[jcsilva]

Ok, no problem... I wait for your feedback.

About the model and software functionalities, I don't know about its future...I just built the docker image for this project: https://github.com/alumae/kaldi-gstreamer-server. Maybe you could try asking there if there will be any improvemente in the future.

[Vishnu-Chittan]

hi jcsilva, Namasthe.... I am trying to run kaldi with gstreamer and docker using the following "sudo docker pull jcsilva/docker-kaldi-gstreamer-server" and I am following the tutorial page " https://medium.com/@nikhilamunipalli/simple-guide-to-kaldi-an-efficient-open-source-speech-recognition-tool-for-extreme-beginners-98a48bb34756 " But I am getting some errors in worker.log file Please guide me how I can solve this issues..... I am herewith attaching the worker.log file also. Dhanyavaad.... worker.log worker.log

python() [0x50160f] PyRun_FileExFlags PyRun_SimpleFileExFlags Py_Main __libc_start_main python() [0x497b8b]

terminate called after throwing an instance of 'std::runtime_error' what():
root@eac4996d259f:/opt# docker run -it -p 8080:80 -m 4g --memory-reserrvation="4g" -v ... bash: docker: command not found root@eac4996d259f:/opt# exit [rrt@rrt ~]$ docker run -it -p 8080:80 -m 4g --memory-reservation="4g" -v ... "docker run" requires at least 1 argument. See 'docker run --help'.

Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container [rrt@rrt ~]$ docker run -it -p 8080:80 -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash root@599466696a02:/opt# clear

root@599466696a02:/opt# ls -l total 8 drwxr-xr-x. 4 root root 79 Jun 13 2018 gst-kaldi-nnet2-online drwxr-xr-x. 5 root root 151 Jun 13 2018 kaldi drwxr-xr-x. 5 root root 247 Jun 13 2018 kaldi-gstreamer-server drwxr-xr-x. 3 root root 39 Dec 19 14:00 models -rwxrwxr-x. 1 root root 992 Jan 30 2016 start.sh -rwxrwxr-x. 1 root root 198 Jan 30 2016 stop.sh root@599466696a02:/opt# ./start.sh -y /opt/models/nnet2.yaml root@599466696a02:/opt# ls -l total 18384 -rw-------. 1 root root 18804736 Dec 25 04:57 core.19678 drwxr-xr-x. 4 root root 79 Jun 13 2018 gst-kaldi-nnet2-online drwxr-xr-x. 5 root root 151 Jun 13 2018 kaldi drwxr-xr-x. 1 root root 28 Jun 13 2018 kaldi-gstreamer-server -rw-r--r--. 1 root root 53 Dec 25 04:57 master.log drwxr-xr-x. 3 root root 39 Dec 19 14:00 models -rwxrwxr-x. 1 root root 992 Jan 30 2016 start.sh -rwxrwxr-x. 1 root root 198 Jan 30 2016 stop.sh -rw-r--r--. 1 root root 6792 Dec 25 04:57 worker.log root@599466696a02:/opt# cat worker.log libdc1394 error: Failed to initialize libdc1394 libudev: udev_has_devtmpfs: name_to_handle_at on /dev: Operation not permitted DEBUG 2018-12-25 04:57:05,756 Starting up worker 2018-12-25 04:57:05 - INFO: decoder2: Creating decoder using conf: {'post-processor': "perl -npe 'BEGIN {use IO::Handle; STDOUT->autoflush(1);} s/(.*)/\1./;'", 'logging': {'version': 1, 'root': {'level': 'DEBUG', 'handlers': ['console']}, 'formatters': {'simpleFormater': {'datefmt': '%Y-%m-%d %H:%M:%S', 'format': '%(asctime)s - %(levelname)7s: %(name)10s: %(message)s'}}, 'disable_existing_loggers': False, 'handlers': {'console': {'formatter': 'simpleFormater', 'class': 'logging.StreamHandler', 'level': 'DEBUG'}}}, 'use-vad': False, 'decoder': {'ivector-extraction-config': 'opt/models/english/tedlium_nnet_ms_sp_online/conf/ivector_extractor.conf', 'num-nbest': 10, 'lattice-beam': 6.0, 'acoustic-scale': 0.083, 'do-endpointing': True, 'beam': 10.0, 'max-active': 10000, 'fst': 'opt/models/english/tedlium_nnet_ms_sp_online/HCLG.fst', 'mfcc-config': 'opt/models/english/tedlium_nnet_ms_sp_online/conf/mfcc.conf', 'use-threaded-decoder': True, 'traceback-period-in-secs': 0.25, 'model': 'opt/models/english/tedlium_nnet_ms_sp_online/final.mdl', 'word-syms': 'opt/models/english/tedlium_nnet_ms_sp_online/words.txt', 'endpoint-silence-phones': '1:2:3:4:5:6:7:8:9:10', 'chunk-length-in-secs': 0.25}, 'silence-timeout': 10, 'out-dir': 'tmp', 'use-nnet2': True} 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: ivector-extraction-config = opt/models/english/tedlium_nnet_ms_sp_online/conf/ivector_extractor.conf 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: num-nbest = 10 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: lattice-beam = 6.0 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: acoustic-scale = 0.083 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: do-endpointing = True 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: beam = 10.0 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: max-active = 10000 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: mfcc-config = opt/models/english/tedlium_nnet_ms_sp_online/conf/mfcc.conf 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: traceback-period-in-secs = 0.25 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: word-syms = opt/models/english/tedlium_nnet_ms_sp_online/words.txt ERROR: SymbolTable::ReadText: Can't open file opt/models/english/tedlium_nnet_ms_sp_online/words.txt 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: endpoint-silence-phones = 1:2:3:4:5:6:7:8:9:10 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: chunk-length-in-secs = 0.25 2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: fst = opt/models/english/tedlium_nnet_ms_sp_online/HCLG.fst ERROR ([5.4.176~1-be967]:Input():kaldi-io.cc:756) Error opening input stream opt/models/english/tedlium_nnet_ms_sp_online/HCLG.fst

[ Stack-Trace: ] kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*) kaldi::MessageLogger::~MessageLogger() kaldi::Input::Input(std::string const&, bool*) fst::ReadFstKaldiGeneric(std::string, bool)

g_object_set_property

PyEval_EvalFrameEx PyEval_EvalFrameEx . . . python() [0x4b988b] PyEval_EvalFrameEx PyEval_EvalFrameEx PyEval_EvalCodeEx python() [0x50160f] PyRun_FileExFlags PyRun_SimpleFileExFlags Py_Main __libc_start_main python() [0x497b8b]

2018-12-25 04:57:05 - INFO: decoder2: Setting decoder property: model = opt/models/english/tedlium_nnet_ms_sp_online/final.mdl ERROR ([5.4.176~1-be967]:Input():kaldi-io.cc:756) Error opening input stream opt/models/english/tedlium_nnet_ms_sp_online/final.mdl

[ Stack-Trace: ] kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*) kaldi::MessageLogger::~MessageLogger() kaldi::Input::Input(std::string const&, bool*)

g_object_set_property

PyEval_EvalFrameEx PyEval_EvalFrameEx python() [0x4e4518] . . . python() [0x4b988b] PyEval_EvalFrameEx PyEval_EvalFrameEx PyEval_EvalCodeEx python() [0x50160f] PyRun_FileExFlags PyRun_SimpleFileExFlags Py_Main __libc_start_main python() [0x497b8b]

2018-12-25 04:57:05 - INFO: decoder2: Created GStreamer elements 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstAppSrc object at 0x7fa0b71dc320 (GstAppSrc at 0x2a5c5b0)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstDecodeBin object at 0x7fa0b71dc2d0 (GstDecodeBin at 0x2b08060)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstAudioConvert object at 0x7fa0b71dc3c0 (GstAudioConvert at 0x2b13ce0)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstAudioResample object at 0x7fa0b71dc280 (GstAudioResample at 0x2900f70)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstTee object at 0x7fa0b71dc370 (GstTee at 0x2b22000)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstQueue object at 0x7fa0b71dc460 (GstQueue at 0x2b24170)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstFileSink object at 0x7fa0b71dc4b0 (GstFileSink at 0x2b28400)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstQueue object at 0x7fa0b71dc500 (GstQueue at 0x2b24460)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.Gstkaldinnet2onlinedecoder object at 0x7fa0b71dc550 (Gstkaldinnet2onlinedecoder at 0x2b42150)> to the pipeline 2018-12-25 04:57:05 - DEBUG: decoder2: Adding <main.GstFakeSink object at 0x7fa0b71dc5a0 (GstFakeSink at 0x2b40210)> to the pipeline 2018-12-25 04:57:05 - INFO: decoder2: Linking GStreamer elements ERROR ([5.4.176~1-be967]:ReadConfigFile():parse-options.cc:469) Cannot open config file: opt/models/english/tedlium_nnet_ms_sp_online/conf/mfcc.conf

[ Stack-Trace: ] kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*) kaldi::MessageLogger::~MessageLogger() kaldi::ParseOptions::ReadConfigFile(std::string const&) void kaldi::ReadConfigFromFilekaldi::MfccOptions(std::string, kaldi::MfccOptions*) kaldi::OnlineNnet2FeaturePipelineInfo::OnlineNnet2FeaturePipelineInfo(kaldi::OnlineNnet2FeaturePipelineConfig const&)

gst_pad_query gst_pad_query_caps gst_element_get_compatible_pad gst_element_link_pads_full . . . python() [0x4b988b] PyEval_EvalFrameEx PyEval_EvalFrameEx PyEval_EvalCodeEx python() [0x50160f] PyRun_FileExFlags PyRun_SimpleFileExFlags Py_Main __libc_start_main python() [0x497b8b]

terminate called after throwing an instance of 'std::runtime_error' what():

[jcsilva]

your error is here:

ERROR: SymbolTable::ReadText: Can't open file opt/models/english/tedlium_nnet_ms_sp_online/words.txt

it seems there is no "opt/models/english/" in your docker container. Please, check your parameters (specially "-v") when you run docker

[Vishnu-Chittan]

@jcsilva Thank you so much for your response. Will you suggest which parameter I can use for it.

[Vishnu-Chittan]

my container has files in opt/models/ here is my container data root@9bdd927a25b8:/opt# cd models root@9bdd927a25b8:/opt/models# ls english nnet2.yaml root@9bdd927a25b8:/opt/models# cd english/ root@9bdd927a25b8:/opt/models/english# ls tedlium_nnet_ms_sp_online root@9bdd927a25b8:/opt/models/english# cd tedlium_nnet_ms_sp_online/ root@9bdd927a25b8:/opt/models/english/tedlium_nnet_ms_sp_online# ls G.carpa G.fst HCLG.fst conf final.mdl ivector_extractor phones.txt word_boundary.int words.txt root@9bdd927a25b8:/opt/models/english/tedlium_nnet_ms_sp_online#