MiniCPM-V icon indicating copy to clipboard operation
MiniCPM-V copied to clipboard

about multi-image finetune

Open univa-JASON opened this issue 1 year ago • 4 comments

when i finetune v2_5, can i compose dataset like this?

[
    {
      "id": "0",
      "image": {
        "<image_00>": "path/to/image_0.jpg",
        "<image_01>": "path/to/image_1.jpg",
        "<image_02>": "path/to/image_2.jpg"
      },
      "conversations": [
        {
          "role": "user", 
          "content": "<image_00>"
        }, 
        {
          "role": "assistant", 
          "content": "main_text"
        }
        {
          "role": "user", 
          "content": "<image_01>"
        }, 
        {
          "role": "assistant", 
          "content": "caption"
        }
        {
          "role": "user", 
          "content": "<image_02>"
        }, 
        {
          "role": "assistant", 
          "content": "caption"
        }
      ]
    }
  ]

univa-JASON avatar Aug 28 '24 02:08 univa-JASON

hi,2.5 can not support multi-image finetune

LDLINGLINGLING avatar Aug 28 '24 03:08 LDLINGLINGLING

oh, thanks for your answer. then only v2_6 support multi image? And when finetune with interleaved dataset, that format can be used for v2_6?

univa-JASON avatar Aug 28 '24 04:08 univa-JASON

yes,you can finetune with interleaved dataset

LDLINGLINGLING avatar Aug 28 '24 06:08 LDLINGLINGLING

i see. thank you so much!!

univa-JASON avatar Aug 28 '24 06:08 univa-JASON