QuickUMLS icon indicating copy to clipboard operation
QuickUMLS copied to clipboard

Solve nested entities problems by using SpanCategorizer

Open hungvo304ml opened this issue 3 years ago • 0 comments

Using doc.spans["sc"] (SpanCategorizer) to solve the problem of overlapped tokens in nested NER for spacy. By replacing doc.ents with doc.spans["sc"], all possible entities are able to be stored without any errors. After storing all possible spans, we filter out overlapping spans before adding them to doc.ents. Here we remove overlapping spans using spacy.util.filter_spans. When spans overlap, the rule is to prefer the first longest span over shorter ones.

hungvo304ml avatar Sep 29 '22 20:09 hungvo304ml