Langchain-Chatchat icon indicating copy to clipboard operation
Langchain-Chatchat copied to clipboard

文本分割的时候,能不能按照txt文件的每行进行分割,也就是按照换行符号\n进行分割???

Open cristianohello opened this issue 1 year ago • 1 comments

下面的代码应该怎么修改?

from langchain.text_splitter import CharacterTextSplitter import re from typing import List

class ChineseTextSplitter(CharacterTextSplitter): def init(self, pdf: bool = False, **kwargs): super().init(**kwargs) self.pdf = pdf

def split_text(self, text: str) -> List[str]:
    if self.pdf:
        text = re.sub(r"\n{3,}", "\n", text)
        text = re.sub('\s', ' ', text)
        text = text.replace("\n\n", "")
    sent_sep_pattern = re.compile('([﹒﹔﹖﹗.。!?]["’”」』]{0,2}|(?=["‘“「『]{1,2}|$))')  # del :;
    sent_list = []
    for ele in sent_sep_pattern.split(text):
        if sent_sep_pattern.match(ele) and sent_list:
            sent_list[-1] += ele
        elif ele:
            sent_list.append(ele)
    return sent_list

cristianohello avatar May 18 '23 08:05 cristianohello