TSH333

I used 4 gpus SOLVER.IMS_PER_BATCH=4 At the beginning of training, the GPU memory usage was 9000m+, but after 9w iter, the memory usage became uneven, and the memory of a...

I haven't found a way to quantify it in the current project.May I have a suggestion

loss 在第二个step就变成了0

5

loss 在第二个step 就变成了0 训练依旧可以继续运行；这个情况 导致我 不能够了解 训练的情况，出现这种情况的原因可能是什么呢，数据的准备我剔除了较大的图像以及较小的图像，训练的图像大小都在448*448附近；并且都是单轮对话 `[ { "id": "identity_0", "conversations": [ { "from": "user", "value": "Picture 1: 5019b9e77382910074.png\ntext1" }, { "from": "assistant", "value": "(222,333),(444,555)" }] ` ` {'loss':...

### 是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this? - [X] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions ### 该问题是否在FAQ中有解答？ | Is there an...

Have you ever merged a Qwen2-VL model before?

14

Traceback (most recent call last): File "/opt/conda/bin/mergekit-yaml", line 8, in sys.exit(main()) ^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1161, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/click/core.py", line 1082, in main rv =...