`Align` using `--reference_directory` flag got error

[xiaozhah]

I want to calculate the FA metrics like this then I refered this link , and found there was a Evaluation mode in align mode usingreference_directory flag. But I got some error even using the MFA output as gold-standard reference, see below

mfa align ./input mandarin_pinyin mandarin ./tg1 --overwrite --clean mfa align ./input mandarin_pinyin mandarin ./tg --overwrite --clean --reference_directory ./tg1

Then I got this, looks like because of SQL?

ERROR - There was an error in the run, please see the log.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: NOT NULL constraint failed: reference_phone_interval.begin

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\Scripts\mfa-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\ProgramData\Miniconda3\lib\site-packages\montreal_forced_aligner\command_line\mfa.py", line 1062, in main
    run_align_corpus(args, unknown)
  File "C:\ProgramData\Miniconda3\lib\site-packages\montreal_forced_aligner\command_line\align.py", line 107, in run_align_corpus
    align_corpus(args, unknown_args)
  File "C:\ProgramData\Miniconda3\lib\site-packages\montreal_forced_aligner\command_line\align.py", line 52, in align_corpus
    aligner.load_reference_alignments(args.reference_directory)
  File "C:\ProgramData\Miniconda3\lib\site-packages\montreal_forced_aligner\corpus\acoustic_corpus.py", line 207, in load_reference_alignments
    conn.execute(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\sql\elements.py", line 333, in _execute_on_connection
    return connection._execute_clauseelement(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\util\compat.py", line 208, in raise_
    raise exception
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "C:\ProgramData\Miniconda3\lib\site-packages\sqlalchemy\engine\default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) NOT NULL constraint failed: reference_phone_interval.begin
[SQL: INSERT INTO reference_phone_interval DEFAULT VALUES]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

[mmcauliffe]

What's the format of the reference alignments? They should be TextGrids like what would be exported from MFA, but it seems like one or more of the phone intervals does not have a start time.

[Approximetal]

What's the format of the reference alignments? They should be TextGrids like what would be exported from MFA, but it seems like one or more of the phone intervals does not have a start time.

I have the same error, and I found here https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/blob/9a1034580502558aabc6b86bbfc52e1b53730395/montreal_forced_aligner/corpus/acoustic_corpus.py#L266 does not return anyting that results reference_intervals empty.

What is the format of reference dictionary? Does it need the same speaker name?

[li-henan]

hi，xiaozhah，could I ask that if you have resolved this problem？

[li-henan]

and have you also can get alignment_analysis.csv？thanks very much if you can reply

[xiaozhah]

Not success to try the reference_directory, so I manually exported the FA results to *.TextGrid and then calculated the mean and std of absolute duration at the phoneme and word level, as well as the hit accuracy with a tolerance of 25ms. (ref: https://arxiv.org/pdf/2203.16838.pdf)

[li-henan]

Thanks for replying，good luck！

[li-henan]

dear friend，have you trained the model in multilingual situation？I want to use this tool in multilingual situation. Thanks for replying very much

[li-henan]

dear friend，have you test NeuFA and compare it with mfa？I want to choose one and use in multilingual situation. Thanks for your helping.

[xiaozhah]

I do not recommend using NeuFA, as it may not consistently perform well. Instead, I suggest training an HMM (MFA) model for each speaker and utilizing wav2vec2 as a feature extractor to predict the phoneme sequence of multilingual situation. By employing backtracking, you can obtain accurate alignment. This approach yields higher accuracy compared to performing MFA on multiple speakers directly. (ref: https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html) I trained a multilingual dataset using this method and achieved accurate results.

[li-henan]

Thanks for your helping. are you Chinese people？could we add friends on WeChat if it's convenient. my WeChat is ：LHN15839379732

[li-henan]

dear friend，do you according to the way in a large part to align in https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html，or only refer partially this method such as getting features by that？Cause I read this page and find it is used for aligning.

Thanks for your replying.Good wishes！