Medical-Relation-Extraction icon indicating copy to clipboard operation
Medical-Relation-Extraction copied to clipboard

Encoding issues in dataset

Open Bernhard-Steindl opened this issue 2 years ago • 0 comments

Hello!

We found that the dataset contains replacement characters, arbitrary question marks and many non-ascii chars. Are you aware of them? Do you still have raw data without this encoding problem?

Best regards, Bernhard

There are around 525 (or more) lines that have non ascii chars or a question mark:

(NLP_2021w) bernhard@Bernhards-MacBook-Pro % ggrep -H --color='auto' -P -n "[^\x00-\xFF]|\?" ground_truth_cause/* ground_truth_treat/* | wc -l       
     525

Below are the 7 lines which contain the replacement character �:

(NLP_2021w) bernhard@Bernhards-MacBook-Pro data % grep -H "�" ground_truth_cause/* ground_truth_treat/*  | wc -l                        
       7
(NLP_2021w) bernhard@Bernhards-MacBook-Pro data % grep -H "�" ground_truth_cause/* ground_truth_treat/*                                 
ground_truth_cause/test.csv:901704;cause;1;1;1;1;1;PRECOCIOUS PUBERTY;88;106;MCCUNE-ALBRIGHT SYNDROME;0;24;MCCUNE-ALBRIGHT SYNDROME (MAS) is a rare disorder characterized by the classic triad of PRECOCIOUS PUBERTY, polyostotic fibrous dysplasia and caf????����鰹�?��?����???��?��????��?�?????��??????��鰹�?�?��?������?����?????��???����?����????��?������???��????��?�?????��?�??�����?����?����?????��?�???????????��鰹�?�?��?����??????��鰹�?�?��?����???����?????��?�?????����??��???��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��?�����鰹?����?����???��??����?�??????����鰹�????������?��?��????��?�??�?��?��??���������?����?����?????��?�??????????����?��?�??��?��?��???��???��??��???��鰹�?�?��?����????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??����活�????��?�??????��?��?�����?��??����?�??????��?��?????��?�?????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??��?��?�??????��?�??�鰹?��???����?????��鰹�?�?��?����?t spots;PRECOCIOUS PUBERTY;MCCUNE-ALBRIGHT SYNDROME;cause of
ground_truth_cause/test.csv:907865;cause;0;-1;-1;-1;-1;MATURITY ONSET DIABETES;105;128;GLIBENCLAMIDE;59;72;A clinical comparison between glipizide (Glibenese �����?��??����?��?�鰹?��???��GLIBENCLAMIDEnclamide (Daonil �����?��??����?��?�鰹?��???�������?����??n tMATURITY ONSET DIABETESy onset diabetes: a controlled double-blind cross-over study;MATURITY ONSET DIABETES;GLIBENCLAMIDE;may treat
ground_truth_cause/train.csv:907865;cause;0;-1;-1;-1;-1;MATURITY ONSET DIABETES;105;128;GLIBENCLAMIDE;59;72;A clinical comparison between glipizide (Glibenese �����?��??����?��?�鰹?��???��GLIBENCLAMIDEnclamide (Daonil �����?��??����?��?�鰹?��???�������?����??n tMATURITY ONSET DIABETESy onset diabetes: a controlled double-blind cross-over study;MATURITY ONSET DIABETES;GLIBENCLAMIDE;may treat
ground_truth_cause/train.csv:901695;cause;0;-1;1;;;MONOCLOL GAMMOPATHY;87;108;MACROGLOBULINEMIA;161;178;Because of the occasiol familial occurrence of other monoclol gammopathies such as MONOCLOL GAMMOPATHY of undetermined significance, multiple myeloma, and MACROGLOBULINEMIA of Waldenstr鐃緒申鐃緒申羝�申��申鐃緒申鐃緒申����渇���申amyloidosis (AL) should be added to the list of immunopathies with a familial predisposition.;MONOCLOL GAMMOPATHY;MACROGLOBULINEMIA;cause of
ground_truth_cause/train.csv:901704;cause;1;1;1;1;1;PRECOCIOUS PUBERTY;88;106;MCCUNE-ALBRIGHT SYNDROME;0;24;MCCUNE-ALBRIGHT SYNDROME (MAS) is a rare disorder characterized by the classic triad of PRECOCIOUS PUBERTY, polyostotic fibrous dysplasia and caf????����鰹�?��?����???��?��????��?�?????��??????��鰹�?�?��?������?����?????��???����?����????��?������???��????��?�?????��?�??�����?����?����?????��?�???????????��鰹�?�?��?����??????��鰹�?�?��?����???����?????��?�?????����??��???��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��?�����鰹?����?����???��??����?�??????����鰹�????������?��?��????��?�??�?��?��??���������?����?����?????��?�??????????����?��?�??��?��?��???��???��??��???��鰹�?�?��?����????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??����活�????��?�??????��?��?�����?��??����?�??????��?��?????��?�?????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??��?��?�??????��?�??�鰹?��???����?????��鰹�?�?��?����?t spots;PRECOCIOUS PUBERTY;MCCUNE-ALBRIGHT SYNDROME;cause of
ground_truth_treat/train.csv:901704;treat;0;-1;-1;-1;-1;PRECOCIOUS PUBERTY;88;106;MCCUNE-ALBRIGHT SYNDROME;0;24;MCCUNE-ALBRIGHT SYNDROME (MAS) is a rare disorder characterized by the classic triad of PRECOCIOUS PUBERTY, polyostotic fibrous dysplasia and caf????����鰹�?��?����???��?��????��?�?????��??????��鰹�?�?��?������?����?????��???����?����????��?������???��????��?�?????��?�??�����?����?����?????��?�???????????��鰹�?�?��?����??????��鰹�?�?��?����???����?????��?�?????����??��???��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��?�����鰹?����?����???��??����?�??????����鰹�????������?��?��????��?�??�?��?��??���������?����?����?????��?�??????????����?��?�??��?��?��???��???��??��???��鰹�?�?��?����????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??����活�????��?�??????��?��?�����?��??����?�??????��?��?????��?�?????��鰹�?�?��?��?�??��?����????��???��????��?�?????����?��?��???��??������???��?�??????��?�??�???����????��?�??��?��?�??????��?�??�鰹?��???����?????��鰹�?�?��?����?t spots;PRECOCIOUS PUBERTY;MCCUNE-ALBRIGHT SYNDROME;disease has finding
ground_truth_treat/train.csv:901695;treat;0;-1;-1;;;MONOCLOL GAMMOPATHY;87;108;MACROGLOBULINEMIA;161;178;Because of the occasiol familial occurrence of other monoclol gammopathies such as MONOCLOL GAMMOPATHY of undetermined significance, multiple myeloma, and MACROGLOBULINEMIA of Waldenstr鐃緒申鐃緒申羝�申��申鐃緒申鐃緒申����渇���申amyloidosis (AL) should be added to the list of immunopathies with a familial predisposition.;MONOCLOL GAMMOPATHY;MACROGLOBULINEMIA;disease has finding

Bernhard-Steindl avatar Oct 31 '21 11:10 Bernhard-Steindl