Lihui Chen

icon company Duke Kunshan University icon location Jiangsu icon location Senior Undergraduate Student in Data Science, Duke Kunshan University. email: [email protected]

Results 1 comments of Lihui Chen

文档超了llm的最大tokens，如何有效处理。

> +1，chunk_text方法直接str(text)了 > > ` def chunk_text(self, text) -> Tuple[List[str], Dict[str, str]]: if self.dataset_name in self.datasets_no_chunk: chunks = [f"{text.get('title', '')} {text.get('text', '')}".strip() if isinstance(text, dict) else str(text)] else: chunks =...