InternEvo
InternEvo copied to clipboard
InternLM
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
# InternLM Simulator ## 1. Introduction The solver mainly consists of two components: 1. `profiling`: Collects the time consumption of each stage during the model training process in advance and...
### Describe the feature MoE模型里稠密层和专家层zero和并行的解耦 ### Will you implement it? - [ ] I would like to implement this feature and create a PR!
### Describe the feature 实际使用过程中，不需要memory_pool，memory pool的逻辑可能和其他芯片的显存分配策略有冲突，建议统一去除memory pool的实现和使用，包括moe对memory pool的使用 ### Will you implement it? - [ ] I would like to implement this feature and create a PR!
### Describe the question.
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand...
### Describe the feature Does InternEvo support fine-tuning using LoRA? ### Will you implement it? - [ ] I would like to implement this feature and create a PR!
### Describe the bug It's a probabilistic occurrence, Socket Timeout when group.allreduce([tensor], opts) if group in _world.pg_coalesce_state.keys(): # We are in coalescing context, do not issue single operation, just append...
### Describe the question. 请问 Internevo是否支持tied_embedding?有的话怎么使用呢？
### Describe the bug 1. 之前给出的脚本仅支持了GShard MoE训练的方式转化hf的脚本，但是如果用MegaBlock进行训练的话权重转换脚本就不适用了。 2. 仍然未提供已经训练好的Internevo的权重转换成internevo MoE权重的脚本。 ### Environment 官方镜像 ### Other information _No response_
### Describe the bug 我来重新描述一下我的问题，我在用internevo训练的时候用的bf16，然后转换成hf后用fp16推理遇到了下述报错 ``` Traceback (most recent call last): File "/InternLM/hf_test.py", line 15, in output = model.generate(**inputs, **gen_kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File...