FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

fix: garbled Chinese in the conv.json

Open DSYZayn opened this issue 1 year ago • 4 comments

Why are these changes needed?

I need save the battle logs in the conv.json, and then I found that garbled Chinese in the file. The reason is json.dumps the ascii encoding used by default during serialization, if you want to output the real Chinese, you need to specify ensure_ascii=False: for more in-depth analysis, it should be that the dJSON object is not a simple unicode implementation, but contains a mixed unicode encoding and a string that has been encoded with utf-8.

Related issue number (if applicable)

Checks

Before fixed: image

After: image

  • [x] I've run format.sh to lint the changes in this PR.
  • [x] I've included any doc changes needed.
  • [x] I've made sure the relevant tests are passing (if applicable).

DSYZayn avatar Mar 10 '24 16:03 DSYZayn

actually @DSYZayn could you double check all the places we dump conversation data to json files are fixed?

infwinston avatar Mar 12 '24 07:03 infwinston

@infwinston Not yet, I only modified the three files that we needed urgently. I'm glad to fix the remain ones, there are many files wait to fix.

DSYZayn avatar Mar 12 '24 07:03 DSYZayn

@infwinston I've fixed all the potential error when dumps data to json file, and I found that httpx.AsyncClient usually cause error when user's network is not good, so I add an environment argument to fix it in openai_api_server.py. But still it will cause the same question when user run gradio_web_server or gradio_web_server_multi, but that's problem of upstream gradio. I suggest user mannually modified in this line. Exact path to the config file is: ~/path-to-venv/lib/python3.9/site-packages/httpx/_config.py ==> line #369

DSYZayn avatar Mar 12 '24 08:03 DSYZayn

@infwinston Excuse me, I noticed it has been double review, took a long time and the bot got a successful check too. Maybe we can think about the possibility of merging it in the main branch😊. As you can see, all changes are basically safe and no aggressiveness.

DSYZayn avatar Apr 15 '24 07:04 DSYZayn