Mismatch between QA-pairs and images in LLaVA-NeXT-Interleave-Bench

[ZJU2021]

I cloned the "lmms-lab/LLaVA-NeXT-Interleave-Bench" dataset from Huggingface. I checked several evaluation samples in it and noticed (possible) mismatch between certain QA-pairs and their associated images.

For example: in "multi_image_in_domain.json", the first sample of the MagicBrush dataset:

{
	"sample_id": 1,
	"conversations": [
		{
			"from": "human",
			"value": "Please provide the image edit instruction that can transfrom the source image to the target image.\nSource Image: <image>\nTarget Image: <image>\nInstruction: "
		},
		{
			"from": "gpt",
			"value": "edit some mountains in the background"
		}
	],
	"image": [
		"Split1/MagicBrush/images/1.jpg",
		"Split1/MagicBrush/images/2.jpg"
	],
	"choice_list": null,
	"metadata": {
		"dataset": "MagicBrush",
		"split": "val",
		"num_sample": 265,
		"task_instruction": "Please provide the image edit instruction that can transfrom the source image to the target image.",
		"question_type": "open-ended"
	}

in "Split1/MagicBrush/images/", "1.jpg" and "2.jpg": (a table -> a dog)

It seems that the QA-pairs and images are mismatched (in the MagicBrush dataset). Would you please check the released dataset and fix such (possible) bugs?