language-learning icon indicating copy to clipboard operation
language-learning copied to clipboard

Published 20 hours ago • verified publisher icon singnet

Using ats (@) and periods (.) for suffixes in Pre-Cleaner, MST-Parser, Grammar Learner and Link Grammar

Open akolonin opened this issue 5 years ago • 3 comments

Few problems:

  1. During iterative grammar learning, tagging words in input corpus and input parses may face ambiguity if the words with ats (@) in parses and corpus are translated to words with Link Grammar (LG) suffixes using period (.).
  2. Emails with inner ats (@) are "corrupted" after Grammar Learner (GL) with changing the ats to periods (.).
  3. There is some problem (TO BE EXPLAINED WITH DATA REFERENCES by @alexei-gl ) about up.'and и up@'and in Grammar Tester (GT).
  4. When the input corpus/parses contain words with ats they are not recognised by GT (because they are stored with periods) which decrease F1 metric.

@glicerico , do you think that using the period (.) in WSD process and nod re-coding periods to ats by GL could eliminate all of the the problems and wouldn't solve other problems in MST-Parsing?

akolonin avatar Mar 28 '19 10:03 akolonin

Item 3 sample is located at http://langlearn.singularitynet.io/data/aglushchenko_parses/suffix-problem/ . The above mentioned token can be easily found in the dictionary file rule.

alexei-gl avatar Apr 05 '19 13:04 alexei-gl

Looks like the problem with up.'and и up@'and is not the akolonin@Ubuntu-1604-xenial-64-minimal:/home/aglushchenko/data/parses/suffix-problem$ grep -P ".'" dict_20C_2019-01-28_0006.4.0.dict | wc -l 1 akolonin@Ubuntu-1604-xenial-64-minimal:/home/aglushchenko/data/parses/suffix-problem$ grep -P "up.'and" dict_20C_2019-01-28_0006.4.0.dict | wc -l 1 grep -P "up.'and" test-corpus-06.txt.raw (dove)(,)(and)(flew)(up.'and)(into)(the)(air)(.)] grep -P ".'" test-corpus-06.txt.raw (dove)(,)(and)(flew)(up.'and)(into)(the)(air)(.)] grep -P ".\w" test-corpus-06.txt.raw | grep -v Found | grep -v Link(the)(man)(at)(the)(other)(end)(of)(them)(..y)] (as)(her)(..y)]

@glicerico - in the version MST-parsed that you are crafting now, can we have MST-Parser configured so it is not breaking words with inner period?

akolonin avatar Apr 05 '19 13:04 akolonin

@akolonin , the new tokenizer-less version of the observer and MST-parser only splits by spaces, so this should not be a problem.

glicerico avatar Apr 11 '19 08:04 glicerico