train 成功了，分享一些细节。 trained successfully! Share some details with guys.

[LYCnight]

train成功了，分享一下环境：

• 系统环境：
  • python==3.11.9
  • transformers==4.33.0
  • pytorch==2.2.0
  • flash-attn==2.6.3
  • ninja==1.11.1.1
  • deepspeed==0.15.0
  • wandb==0.17.8
  • /glm-4-9b 目录下的 modeling_chatglm.py和 tokenization_chatglm.py 都已经替换
  • 在 /ds_config/stage3.json 中设置 "stage3_prefetch_bucket_size": 15099494
  • 在 /glm-4-9b/config.json 中设置 "seq_length": 131072
• general.json: (仅供测试)

{"messages":[{"role": "user", "content": "北京的天气怎么样？"}, {"role": "assistant", "content": "今天北京的天气晴朗，气温在25度左右。"}]}
{"messages":[{"role": "user", "content": "如何在Java中实现单例模式？"}, {"role": "assistant", "content": "你可以使用懒汉式或饿汉式来实现单例模式。以下是饿汉式的示例代码：\n```java\npublic class Singleton {\n    private static final Singleton instance = new Singleton();\n    private Singleton() {}\n    public static Singleton getInstance() {\n        return instance;\n    }\n}\n```"}]}
{"messages":[{"role": "user", "content": "你知道中国的四大发明是什么吗？"}, {"role": "assistant", "content": "中国的四大发明是造纸术、指南针、火药和印刷术。"}]}
{"messages":[{"role": "user", "content": "有什么推荐的中文书籍吗？"}, {"role": "assistant", "content": "如果你喜欢历史类书籍，我推荐《资治通鉴》。如果你对文学感兴趣，可以看看《红楼梦》。"}]}
{"messages":[{"role": "user", "content": "What's the weather like in New York?"}, {"role": "assistant", "content": "The weather in New York is cloudy with a temperature around 18°C."}]}
{"messages":[{"role": "user", "content": "How can I implement a stack in Python?"}, {"role": "assistant", "content": "You can implement a stack using a list in Python. Here’s a simple example:\n```python\nstack = []\nstack.append(1)\nstack.append(2)\nprint(stack.pop())  # Outputs 2\n```"}]}
{"messages":[{"role": "user", "content": "Can you explain the concept of machine learning?"}, {"role": "assistant", "content": "Machine learning is a field of artificial intelligence where algorithms learn from data to make predictions or decisions without being explicitly programmed."}]}
{"messages":[{"role": "user", "content": "What's a good book to learn about AI?"}, {"role": "assistant", "content": "A great book to start with is 'Artificial Intelligence: A Modern Approach' by Stuart Russell and Peter Norvig. It covers a wide range of AI topics comprehensively."}]}

• tips:

  • /glm-4-9b 目录下的 modeling_chatglm.py和 tokenization_chatglm.py 都已经替换

    这一步不确定自己是否成功的小伙伴，可以在这两个代码中自己加一些注释，如果报错了就检查一下报错代码中有没有这些注释。