[BUG] Bad tokenizer for Chinese

[Liu-Angelo]

User: 量化学习CAT策略了解多少

ChatGPT: 对不起，您说的语言与我的训练语言不同。您是否需要翻译或其他帮助？

User: 量化交易CTA策略详细介绍一下

ChatGPT: CTA (Commodity Trading Advisor) 策略是指那些以大宗商品为基础的交易策略。这些策略通常使用自动化技术，如计算机程序和数学模型，以识别交易机会并执行交易。

CTA策略可以分为两类：短线交易策略和长线交易策略。短线交易策略通常以高频交易为特点，试图在短期内获利；长线交易策略则试图在较长时间内获利。

CTA策略可以适用于各种大宗商品，如农产品、金属、能源等。它们的目标是通过对市场价格波动的分析和预测，识别交易机会并执行交易。

总的来说，CTA策略是一种高风险、高回报的投资策略，适合对投资风险有承受能力和对市场行情有较深了解的投资者。

User: 高频交易怎么理解

ChatGPT: Traceback (most recent call last): File "/usr/local/bin/OfficialChatGPT", line 8, in sys.exit(main()) File "/usr/local/lib/python3.6/site-packages/revChatGPT/Official.py", line 419, in main for response in chatbot.ask_stream(prompt): File "/usr/local/lib/python3.6/site-packages/revChatGPT/Official.py", line 84, in ask_stream stream=True, File "/usr/local/lib/python3.6/site-packages/openai/api_resources/completion.py", line 31, in create return super().create(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 67, in create "post", url, params, headers, stream=stream File "/usr/local/lib/python3.6/site-packages/openai/api_requestor.py", line 129, in request resp = self.interpret_response(rbody, rcode, rheaders, stream=stream) File "/usr/local/lib/python3.6/site-packages/openai/api_requestor.py", line 348, in interpret_response return self.interpret_response_line(rbody, rcode, rheaders, stream) File "/usr/local/lib/python3.6/site-packages/openai/api_requestor.py", line 368, in interpret_response_line rbody, rcode, resp.data, rheaders, stream_error=stream_error openai.error.InvalidRequestError: This model's maximum context length is 4097 tokens, however you requested 4565 tokens (798 in your prompt; 3767 for the completion). Please reduce your prompt; or completion length.

[acheong08]

What version you using?

[acheong08]

Also, please use Python3.10.