lexical_unit_self_info calculate

[lizijian-buaa]

Dear Yucheng Li: For the condition that a token overlaps multiple lexical_unit, we assign its self_info divided by the overlap number to each overlapped lexical_units. Is my understanding right? https://github.com/liyucheng09/Selective_Context/blob/3074343653bbf3559a87a588667e843744bc6f2a/context_manager.py#L240C21-L242C24 Should we use numerical average instead of weighted average in such case? https://github.com/liyucheng09/Selective_Context/blob/3074343653bbf3559a87a588667e843744bc6f2a/context_manager.py#L248 Thx!