Struggling to detect English

[peterbe]

I took your awesome lib and wrapped it in a little command line app. I also added a conversion table from ISO 639-3 to ISO 639-1.

▶ ./langdetect "Le candidat socialiste à l’élection présidentielle"
Language: fra Script: Latin
fr

Correct!

▶ ./langdetect "Mitt namn på svenska är Peter"
Language: swe Script: Latin
sv

Correct!

But....

▶ ./langdetect "testing in english"
Language: uig Script: Latin
ug

Not right.

▶ ./langdetect "wondering if it still works in English"
Language: nld Script: Latin
nl

Not right either.

Also, would it be possible to output a list of probabilities? That way my app, where I hope to use this, could throw warnings if the probabilities "aren't certain enough".

[peterbe]

By the way, my hack is here: https://github.com/peterbe/langdetect Careful, I haven't done go for a long time.

[abadojack]

I noticed this while I was testing and I haven't found a solution yet. I added it to the TODO list in this commit.

[peterbe]

For what it's worth, I tested something similar with a different project (https://pypi.python.org/pypi/guess_language-spirit) and it's based on a bunch of trigrams too. I can't remember the example but it suffered the same problem.

[abadojack]

I just checked it out and tested it with the same phrases you used.

>>> guess_language("Le candidat socialiste à l’élection présidentielle.")
'fr'

Correct!

>>> guess_language("Mitt namn på svenska är Peter")
'sv'

Correct!

>>> guess_language("testing in english")
'UNKNOWN'

Not correct.

>>> guess_language("wondering still if it works in english")
'af'

Still not correct.

Considering the trigrams used in that project are not similar to the ones I used ... wtf is wrong with English ?

[peterbe]

I tried franc too.

var franc = require('franc');

console.log(franc("Le candidat socialiste à l’élection présidentielle."))
console.log(franc("Mitt namn på svenska är Peter"))
console.log(franc("testing in english"))
console.log(franc("wondering still if it works in english"))

output;

▶ node test.js
fra
swe
uig
nld

[azer]

Short inputs like "happy" "hello" also returns incorrect results.

[ernsheong]

Looks like the longer the string, the better the result?

Hilarious:

func TestLangDetection(t *testing.T) {
	lang := whatlanggo.DetectLang("english english english english english english english")
	if lang != whatlanggo.Eng {
		t.Fatalf("Expected lang to be %v but was %v", whatlanggo.LangToString(whatlanggo.Eng), whatlanggo.LangToString(lang))
	}
}
 
--- FAIL: TestLangDetection (0.00s)
        search_test.go:12: Expected lang to be eng but was uzb
FAIL

Providing a Whitelist via DetectLangWithOptions helps.

Algorithm:

1. http://greyblake.com/blog/2017/07/30/introduction-to-rust-whatlang-library-and-natural-language-identification-algorithms/
2. http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf

Quote from the article:

Disadvantages:

May provide falsy results for short texts (smaller than 200-300 letters). Whatlang tries to compensate this with is_reliable attribute.

So there are 2 ways to fix/mitigate this:

1. Provide confidence value
2. As the article says, for lower amount of text we could try dictionary checks instead.

[miku]

First, thanks for making this library available, it helped me to get rid of a C dependency from cld2.

Just wanted to add a few additional data points concerning non-detection of English text:

• "We report on 630 nm-band AIGalnP strained MQW laser diodes incorporating an MQB. The laser offer high-temperature operation over 60/spl deg/C and have been operating reliably for more than 1,000 h under 3 mW at 45/spl deg/C." -- dan
• "Transverse-mode stabilized GaInP/AlGaInP strained multiquantum well lasers emitting at 638 nm were grown on a 15 degrees off" -- dan
• "A convenient synthesis of vicinal methoxychlorides, methoxyiodides from alkenes using diphenyldiiodo-tetrachloride/methanol, iodine/diphenyldiiodotetra-chloride/methanol, iodine/3-carboxyphenyliod odichloride/methanol is described." -- spa
• "We demonstrate ultra-wideband (850 to 1550 nm) WDM transmission in multi-mode fiber by using single-mode photonic crystal fiber (PCF) as center launching and mode-filtering devices." -- deu

[greyblake]

Hi. I am a creator of the original library in Rust. I just want to say, that you should not have high expectation, if input text is relatively small. The library is based on statistical profiles of languages (trigrams). The bigger input text, the better it represents its statistical profile.

There is nothing you can do with this issue.

Some possible solutions however:

• Use whitelist / blacklist
• Use another library (possible with combination of this one) that is fundamentally different (.e.g based on vocabulary)

@miku

Just wanted to add a few additional data points concerning non-detection of English text:

Rust version at the moment provides is_reliable boolean in result. When it's true it is guaranteed, that language is recognized correctly. Otherwise you should not trust the result. For all of your text samples it returns result is_reliable=false.

[mmorells]

#15 Added confidence with the logic from the original library by @greyblake Doesn't fix the problems with detection, but now Info{} has the confidence rating.