PyWxDump icon indicating copy to clipboard operation
PyWxDump copied to clipboard
verified publisher icon xaoyaoo

导出过大聊天至csv时遇到请求失败

Open OvercloudX opened this issue 6 months ago • 5 comments

问题描述 在尝试导出csv文件时，消息数量非常多的聊天（大概2000多页）无法导出，网页ui中显示请求失败[object: Object] 控制台显示can not unpack non-iterable NoneType object 而消息数很少的聊天可以成功导出

复现步骤 请提供重现问题所需的步骤。（执行的命令）

  1. 打开微信
  2. 迁移聊天记录从手机到电脑
  3. 打开cmd，通过 -h启动后，再通过（端口5000）进入web图形界面
  4. 选择聊天，实时消息，导出，csv文件
  5. 请求失败

预期行为 预期成功导出得到csv文件

实际行为 出现请求失败，控制台出现

Image

环境信息

  • pywxdump版本：Release中 wxdemp.exe 3月10日版本
  • 操作系统版本：win10
  • python版本：
  • 微信版本：

其他信息 请提供任何与问题相关的其他信息（文字，截图等）。

OvercloudX avatar Jun 03 '25 13:06 OvercloudX

个人预测要么是聊天过大要么是聊天中有不可解析的文件格式

OvercloudX avatar Jun 03 '25 13:06 OvercloudX

麻烦把控制台显示的内容发我看看

xaoyaoo avatar Jun 04 '25 15:06 xaoyaoo

@xaoyaoo 同2000页聊天记录也遇到了这个问题，看起来是地图消息解析错误？在wxdump.log找到的报错如下： [E] 2025-06-05 01:32:05 [db_prepare:40] D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\utils\common_utils.py:34 db_error: 'x' Traceback (most recent call last): File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\utils\common_utils.py", line 32, in wrapper return func(*args, **kwargs) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\dbMSG.py", line 274, in get_msg_detail msg = (f"γ : {location.pop('x')} : {location.pop('y')} \n" KeyError: 'x' [E] 2025-06-05 01:32:05 [db_prepare:40] D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\utils\common_utils.py:34 db_error: 'NoneType' object is not subscriptable Traceback (most recent call last): File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\utils\common_utils.py", line 32, in wrapper return func(*args, **kwargs) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\dbMSG.py", line 115, in get_msg_list wxid_list = {d['talker'] for d in rdata} # һ ظ wxid б File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db\dbMSG.py", line 115, in wxid_list = {d['talker'] for d in rdata} # һ ظ wxid б TypeError: 'NoneType' object is not subscriptable [E] 2025-06-05 01:32:05 [uvicorn.error:40] D:\Anaconda\envs\wx_analyse\lib\site-packages\uvicorn\protocols\http\h11_impl.py:408 Exception in ASGI application Traceback (most recent call last): File "D:\Anaconda\envs\wx_analyse\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] File "D:\Anaconda\envs\wx_analyse\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 60, in call return await self.app(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\fastapi\applications.py", line 1054, in call await super().call(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\applications.py", line 112, in call await self.middleware_stack(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\middleware\errors.py", line 187, in call raise exc File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\middleware\errors.py", line 165, in call await self.app(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\middleware\cors.py", line 93, in call await self.simple_response(scope, receive, send, request_headers=headers) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\middleware\cors.py", line 144, in simple_response await self.app(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\middleware\exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app raise exc File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\routing.py", line 715, in call await self.middleware_stack(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\routing.py", line 735, in app await route.handle(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\routing.py", line 288, in handle await self.app(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app raise exc File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\routing.py", line 73, in app response = await f(request) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\fastapi\routing.py", line 301, in app raw_response = await run_endpoint_function( File "D:\Anaconda\envs\wx_analyse\lib\site-packages\fastapi\routing.py", line 214, in run_endpoint_function return await run_in_threadpool(dependant.call, **values) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\starlette\concurrency.py", line 37, in run_in_threadpool return await anyio.to_thread.run_sync(func) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "D:\Anaconda\envs\wx_analyse\lib\site-packages\anyio_backends_asyncio.py", line 2364, in run_sync_in_worker_thread return await future File "D:\Anaconda\envs\wx_analyse\lib\site-packages\anyio_backends_asyncio.py", line 864, in run result = context.run(func, *args) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\api\remote_server.py", line 436, in get_export_csv code, ret = export_csv(wxid, outpath, db_config, my_wxid=my_wxid) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\api\export\exportCSV.py", line 40, in export_csv data, users_t = db.get_msgs(wxid, start_index, page_size) File "D:\Anaconda\envs\wx_analyse\lib\site-packages\pywxdump\db_init.py", line 62, in get_msgs msgs0, wxid_list0 = self.get_msg_list(wxids=wxids, start_index=start_index, page_size=page_size, TypeError: cannot unpack non-iterable NoneType object

Thomas333333 avatar Jun 04 '25 17:06 Thomas333333

@xaoyaoo 我将聊天记录缩减到近8个月就导出成功了，暂时没办法再复现控制台消息了不好意思。 @Thomas333333 我不确认是否是地图信息，但最后显示的TypeError: cannot unpack non-iterable NoneType object是一样的

OvercloudX avatar Jun 04 '25 18:06 OvercloudX

https://github.com/xaoyaoo/PyWxDump/issues/230#issuecomment-2940904556 关于我之前说的地图消息解析错误，我试着debug： 1.在pywxdump\db\dbMSG.py的273行加入print(f"地图消息内容: {StrContent}")，发现以下内容的StrContent导致出错

wxid_123:
<msg>
        <location x="123" y="123" scale="1" label="123" maptype="roadmap" poiname="123" poiid="" buildingId="" floorName="" fromusername="wxid_123" />
</msg>

类似的还有

123@openim:
<msg>
        <location x="123" y="123" scale="1" label="123" maptype="roadmap" poiname="123" poiid="" buildingId="" floorName="" poiCategoryTips="" poiBusinessHour="" poiPhone="" poiPriceTips="" isFromPoiList="" adcode="" cityname="" fromusername="123@openim" />
</msg>

猜想是<msg>前面的wxid_xxx:影响了xml解析。 2.在pywxdump\db\dbMSG.py的274行加入StrContent = re.sub(r'^.*(<msg.*)', r'\1', StrContent, flags=re.DOTALL)，用来去除<msg>前面的字符串，重新运行之后导出成功。

希望能有所帮助

Thomas333333 avatar Jun 05 '25 10:06 Thomas333333