Mengwei Zhang

不能处理parquet文件，想增加对parquet文件处理功能，怎么修改代码呢？

cannot import name 'AutoTokenizer' from 'transformers'错误

1

cannot import name 'AutoTokenizer' from 'transformers' (/home/zwm/miniconda3/envs/Langchain-Chatchat2/lib/python3.11/site-packages/transformers/__init__.py) cannot import name 'AutoTokenizer' from 'transformers' (/home/zwm/miniconda3/envs/Langchain-Chatchat2/lib/python3.11/site-packages/transformers/__init__.py)cannot import name 'AutoTokenizer' from 'transformers' (/home/zwm/miniconda3/envs/Langchain-Chatchat2/lib/python3.11/site-packages/transformers/__init__.py) cannot import name 'AutoTokenizer' from 'transformers' (/home/zwm/miniconda3/envs/Langchain-Chatchat2/lib/python3.11/site-packages/transformers/__init__.py) cannot import name...

使用milvus数据库，初始化数据库的时候，显存爆炸。

1

This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() return self.fget.__get__(instance, owner)() Batches: 0%| | 0/396 [00:00

切换milvus向量库，初始化数据报错

1

Traceback (most recent call last): File "/home/zwm/Code_Program/Chatchat/milvus-Langchain-Chatchat/init_database.py", line 107, in folder2db(kb_names=args.kb_name, mode="recreate_vs", embed_model=args.embed_model) File "/home/zwm/Code_Program/Chatchat/milvus-Langchain-Chatchat/server/knowledge_base/migrate.py", line 128, in folder2db files2vs(kb_name, kb_files) File "/home/zwm/Code_Program/Chatchat/milvus-Langchain-Chatchat/server/knowledge_base/migrate.py", line 113, in files2vs kb.add_doc(kb_file=kb_file, not_refresh_vs_cache=True) File...

ERROR: RPC error: [batch_insert], <ParamError: (code=1, message=expect string input, got: <class 'int'>)>, <Time:{'RPC start': '2024-05-13 01:07:12.372386', 'RPC error': '2024-05-13 01:07:12.372994'}>

1

2024-05-13 01:07:12,373 - decorators.py[line:146] - ERROR: RPC error: [batch_insert], , 2024-05-13 01:07:12,373 - milvus.py[line:595] - ERROR: Failed to insert batch starting at entity: 0/25499 Traceback (most recent call last): File...

ERROR: RPC error: [batch_insert], <ParamError: (code=1, message=invalid input, length of string exceeds max length. length: 131239, max length: 65535)>, <Time:{'RPC start': '2024-05-11 14:17:48.677254', 'RPC error': '2024-05-11 14:17:48.678019'}>

3

2024-05-11 14:17:48,678 - decorators.py[line:146] - ERROR: RPC error: [batch_insert], , 2024-05-11 14:17:48,678 - milvus.py[line:595] - ERROR: Failed to insert batch starting at entity: 4000/11304 Traceback (most recent call last): File...

pymilvus.exceptions.ParamError: <ParamError: (code=1, message=invalid input, length of string exceeds max length. length: 68919, max length: 65535)>

1

2024-05-09 13:37:57,767 - decorators.py[line:146] - ERROR: RPC error: [batch_insert], , 2024-05-09 13:37:57,767 - milvus.py[line:595] - ERROR: Failed to insert batch starting at entity: 2000/37433 Traceback (most recent call last): File...

文件比较多时，要运行几十个小时，一旦出错或者意外终止，数据库就损害了要重新生成。这里每处理100个重新保存一下

2

文件比较多时，要运行几十个小时，一旦出错或者意外终止，数据库就损害了要重新生成。在哪里修改代码，能够每处理100个重新保存一下或者多线程添加到向量数据库。