ottokart
Cannot replicate paper results
Downloading the pre-trained INTERSPEECH-T-BRNN.pcl model from here and then running it on the TED talk data does not yield the reported overall F1 score of 63.1. Is this a reason for this - or am I doing something wrong with the training data?
Below are the obtained metrics, from error_calculator.py
| Punctuation | P | R | F1 |
|---|---|---|---|
| COMMA | 43.9 | 56.0 | 49.2 |
| PERIOD | 61.4 | 70.4 | 65.6 |
| QUESTIONMARK | 50.0 | 62.2 | 55.4 |
| OVERALL | 52.4 | 63.5 | 57.4 |
Here is a sample of the test data:
i'm a savant ,COMMA or more precisely ,COMMA a high-functioning autistic savant .PERIOD it's a rare condition .PERIOD and rarer still when accompanied ,COMMA as in my case ,COMMA by self-awareness and a mastery of language .PERIOD very often when i meet someone and they learn this about me there's a certain kind of awkwardness .PERIOD i can see it in their eyes .PERIOD they want to ask me something .PERIOD and in the end ,COMMA quite often ,COMMA the urge is stronger than they are and they blurt it out ,COMMA if i give you my date of birth ,COMMA can you tell me what day of the week i was born on ?QUESTIONMARK or they mention cube roots or ask me to recite a long number or long text .PERIOD
And here is a sample of the output:
i'm a savant or more precisely a high-functioning autistic savant it's ,COMMA a rare condition and rarer still when accompanied as in my case by self-awareness and a mastery of language ,COMMA very often when i meet someone and they learn this about me ,COMMA there's a certain kind of awkwardness .PERIOD i can see it in their eyes .PERIOD they want to ask me something .PERIOD and in the end ,COMMA quite often ,COMMA the urge is stronger than they are .PERIOD and they blurt it out .PERIOD if i give you my date of birth ,COMMA can you tell me what day of the week i was born on or they mention cube roots or ask me to recite a long number or long text ?QUESTIONMARK
As a side note, attempting to run INTERSPEECH-T-BRNN-pre.pcl gives a ValueError: cannot reshape array of size 10000 into shape (200,256) error, which does not occur when running INTERSPEECH-T-BRNN.pcl or Demo-Europarl-EN.pcl.
Hi,
are you using the https://github.com/ottokart/punctuator2/releases/tag/v1.0 or the latest version?
