ml-ferret
ml-ferret copied to clipboard
apple
Fix typo in CONTRIBUTING.md
The training/inference/eval code has been released. FerretBench Evaluation is also included here. But training data and checkpoints seem not ready yet. Let’s stay tuned!
@Hxyou Hi, great work. Would you like to release the checkpoint files soon? I can't wait to try it~
why this method can boost the grouding ability? I think the "hybrid region representation" can just enhance refering abilitys... who can explain it? thanks
Thank you for your patience. Ckpts have been released last week! Feel free to try it~
why this method can boost the grouding ability? I think the "hybrid region representation" can just enhance refering abilitys... who can explain it? thanks
I think we didn't claim that "hybrid region representation" can help grounding. But instead, in experiments, we ablate whether referring data/task can help grounding when jointly trained, and the answer is yes. I hypothesized that it's because those two tasks all require fine-grained spatial understanding. I.e., by training on one task, the LLM implicitly learns the projection of coordinates and region features to the real location in the image, and thus the other task can also get boosted.
Why not open the issues?
I am sorry for that. The repo was set up by the company. Feel free to email us or leave comments in this pull request as if raising issues. We will try our best to answer.
Hi, thanks for sharing the great work! But why not opening issues? But anyway, I tried to reproduce the ferret-7b evaluation results on refcocog, but my reproduced results are pretty bad, 5% vs 84%.
I exactly followed the instruction steps. The vicuna-7b model I used is https://huggingface.co/lmsys/vicuna-7b-v1.3, and the annotation json is from https://huggingface.co/GLIPModel/GLIP/tree/66ee3ae9a3b8cee0cf78f10ef5fc9a3725db02a1/mdetr_annotations. I manually checked the bbox from the produced 0_of_1.jsonl, most of the boxes are wrong. So it is not the problem of eval_refexp.py (by the way, misc line 23 is missing there). I didn’t encounter any issue or error during installing or checkpoint generation steps, so any suggestion what might be the problem causing the mismatching?
Looking forward to the reply. Thanks for sharing this excellent work again!
Hi, thanks for sharing the great work! But why not opening issues? But anyway, I tried to reproduce the ferret-7b evaluation results on refcocog, but my reproduced results are pretty bad, 5% vs 84%.
I exactly followed the instruction steps. The vicuna-7b model I used is https://huggingface.co/lmsys/vicuna-7b-v1.3, and the annotation json is from https://huggingface.co/GLIPModel/GLIP/tree/66ee3ae9a3b8cee0cf78f10ef5fc9a3725db02a1/mdetr_annotations. I manually checked the bbox from the produced 0_of_1.jsonl, most of the boxes are wrong. So it is not the problem of eval_refexp.py (by the way, misc line 23 is missing there). I didn’t encounter any issue or error during installing or checkpoint generation steps, so any suggestion what might be the problem causing the mismatching?
Looking forward to the reply. Thanks for sharing this excellent work again!
Hi @peiwang062 , thanks for the questions. The visualization of the boxes from the .jsonl file is incorrect because the raw prediction file does not exactly match the image size. The coordinates reflected here are the ones in the range between 0 and 999. In our eval_refexp.py, we do that resize (https://github.com/apple/ml-ferret/blob/262a943e1f856c52d6491de10f443d804b8132ee/ferret/eval/eval_refexp.py#L59) to map back to the original image size. Could you please check that again?
Hi, thanks for sharing the great work! But why not opening issues? But anyway, I tried to reproduce the ferret-7b evaluation results on refcocog, but my reproduced results are pretty bad, 5% vs 84%. I exactly followed the instruction steps. The vicuna-7b model I used is https://huggingface.co/lmsys/vicuna-7b-v1.3, and the annotation json is from https://huggingface.co/GLIPModel/GLIP/tree/66ee3ae9a3b8cee0cf78f10ef5fc9a3725db02a1/mdetr_annotations. I manually checked the bbox from the produced 0_of_1.jsonl, most of the boxes are wrong. So it is not the problem of eval_refexp.py (by the way, misc line 23 is missing there). I didn’t encounter any issue or error during installing or checkpoint generation steps, so any suggestion what might be the problem causing the mismatching? Looking forward to the reply. Thanks for sharing this excellent work again!
Hi @peiwang062 , thanks for the questions. The visualization of the boxes from the .jsonl file is incorrect because the raw prediction file does not exactly match the image size. The coordinates reflected here are the ones in the range between 0 and 999. In our eval_refexp.py, we do that resize (
https://github.com/apple/ml-ferret/blob/262a943e1f856c52d6491de10f443d804b8132ee/ferret/eval/eval_refexp.py#L59
) to map back to the original image size. Could you please check that again?
Hi Haotian, thanks for the quick reply!
For eval.refexp.py, because misc doesn’t exist. I used from torch vision.ops import box_iou to replace your original box_iou and modified Line 168,169 to
iou = box_iou(predict_boxes, target_bbox)
mean_iou = box_iou(predict_boxes.mean(0).view(-1, 4), target_bbox)
Other than, I didn’t do anything. Not sure if the problem is here.
Hi, thanks for sharing the great work! But why not opening issues? But anyway, I tried to reproduce the ferret-7b evaluation results on refcocog, but my reproduced results are pretty bad, 5% vs 84%. I exactly followed the instruction steps. The vicuna-7b model I used is https://huggingface.co/lmsys/vicuna-7b-v1.3, and the annotation json is from https://huggingface.co/GLIPModel/GLIP/tree/66ee3ae9a3b8cee0cf78f10ef5fc9a3725db02a1/mdetr_annotations. I manually checked the bbox from the produced 0_of_1.jsonl, most of the boxes are wrong. So it is not the problem of eval_refexp.py (by the way, misc line 23 is missing there). I didn’t encounter any issue or error during installing or checkpoint generation steps, so any suggestion what might be the problem causing the mismatching? Looking forward to the reply. Thanks for sharing this excellent work again!
Hi @peiwang062 , thanks for the questions. The visualization of the boxes from the .jsonl file is incorrect because the raw prediction file does not exactly match the image size. The coordinates reflected here are the ones in the range between 0 and 999. In our eval_refexp.py, we do that resize ( https://github.com/apple/ml-ferret/blob/262a943e1f856c52d6491de10f443d804b8132ee/ferret/eval/eval_refexp.py#L59
) to map back to the original image size. Could you please check that again?
Hi Haotian, thanks for the quick reply! For
eval.refexp.py, because misc doesn’t exist. I usedfrom torch vision.ops import box_iouto replace your originalbox_iouand modified Line 168,169 toiou = box_iou(predict_boxes, target_bbox) mean_iou = box_iou(predict_boxes.mean(0).view(-1, 4), target_bbox)Other than, I didn’t do anything. Not sure if the problem is here.
Thanks for the findings! As the company is on holiday these days, I will ask folks to recover the missing files once they are back. In the meantime, I will follow up with you about this file in the email. Thanks a lot for the help!
Hi, thanks for sharing the great work! But why not opening issues? But anyway, I tried to reproduce the ferret-7b evaluation results on refcocog, but my reproduced results are pretty bad, 5% vs 84%. I exactly followed the instruction steps. The vicuna-7b model I used is https://huggingface.co/lmsys/vicuna-7b-v1.3, and the annotation json is from https://huggingface.co/GLIPModel/GLIP/tree/66ee3ae9a3b8cee0cf78f10ef5fc9a3725db02a1/mdetr_annotations. I manually checked the bbox from the produced 0_of_1.jsonl, most of the boxes are wrong. So it is not the problem of eval_refexp.py (by the way, misc line 23 is missing there). I didn’t encounter any issue or error during installing or checkpoint generation steps, so any suggestion what might be the problem causing the mismatching? Looking forward to the reply. Thanks for sharing this excellent work again!
Hi @peiwang062 , thanks for the questions. The visualization of the boxes from the .jsonl file is incorrect because the raw prediction file does not exactly match the image size. The coordinates reflected here are the ones in the range between 0 and 999. In our eval_refexp.py, we do that resize ( https://github.com/apple/ml-ferret/blob/262a943e1f856c52d6491de10f443d804b8132ee/ferret/eval/eval_refexp.py#L59
) to map back to the original image size. Could you please check that again?
Hi Haotian, thanks for the quick reply! For
eval.refexp.py, because misc doesn’t exist. I usedfrom torch vision.ops import box_iouto replace your originalbox_iouand modified Line 168,169 toiou = box_iou(predict_boxes, target_bbox) mean_iou = box_iou(predict_boxes.mean(0).view(-1, 4), target_bbox)Other than, I didn’t do anything. Not sure if the problem is here.
Thanks for the findings! As the company is on holiday these days, I will ask folks to recover the missing files once they are back. In the meantime, I will follow up with you about this file in the email. Thanks a lot for the help!
Thank you so much Haotian! Merry Christmas!
Sorry to ask here, but I need some help.
I've followed the issues but am still encountering the same error:
NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE.
Sorry to ask here, but I need some help.
I've followed the [issues](https://github.com/lm-sys/FastChat/issues/412) but am still encountering the same error:
NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE.
Hi @bensonbs , the error might be triggered by multiple reasons. Can you first double-check you can successfully run those three commands (controller, gradio web server, model worker) as instructed in the readme without errors? Then can you show us screenshots of those three programs launched by the three commands when the error in demo happens? This can help to debug where the problem comes from.
Sorry to ask here, but I need some help.
NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE.Hi @bensonbs , the error might be triggered by multiple reasons. Can you first double-check you can successfully run those three commands (controller, gradio web server, model worker) as instructed in the readme without errors? Then can you show us screenshots of those three programs launched by the three commands when the error in demo happens? This can help to debug where the problem comes from.
Here is the result. Thank you for your assistance.
(ferret) root@58dfc909b9e7:/share/ml-ferret# python -m ferret.serve.controller --host 0.0.0.0 --port 10000
[2023-12-28 00:46:15,431] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2023-12-28 00:46:15 | INFO | controller | args: Namespace(host='0.0.0.0', port=10000, dispatch_method='shortest_queue')
2023-12-28 00:46:15 | INFO | controller | Init controller
2023-12-28 00:46:16 | ERROR | stderr | INFO: Started server process [412]
2023-12-28 00:46:16 | ERROR | stderr | INFO: Waiting for application startup.
2023-12-28 00:46:16 | ERROR | stderr | INFO: Application startup complete.
2023-12-28 00:46:16 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:10000 (Press CTRL+C to quit)
2023-12-28 00:46:20 | INFO | controller | Receive unknown heart beat. http://localhost:40000
2023-12-28 00:46:20 | INFO | stdout | INFO: 127.0.0.1:58750 - "POST /receive_heart_beat HTTP/1.1" 200 OK
2023-12-28 00:46:20 | INFO | controller | Register a new worker: http://localhost:40000
2023-12-28 00:46:20 | INFO | controller | Register done: http://localhost:40000, {'model_names': ['FERRET-13B-v0'], 'speed': 1, 'queue_length': 0}
2023-12-28 00:46:20 | INFO | stdout | INFO: 127.0.0.1:58764 - "POST /register_worker HTTP/1.1" 200 OK
2023-12-28 00:46:35 | INFO | controller | Receive heart beat. http://localhost:40000
2023-12-28 00:47:23 | INFO | controller | Register a new worker: http://localhost:40000
2023-12-28 00:47:23 | INFO | controller | Register done: http://localhost:40000, {'model_names': ['FERRET-13B-v0'], 'speed': 1, 'queue_length': 0}
2023-12-28 00:47:23 | INFO | stdout | INFO: 127.0.0.1:45306 - "POST /refresh_all_workers HTTP/1.1" 200 OK
2023-12-28 00:47:23 | INFO | stdout | INFO: 127.0.0.1:45312 - "POST /list_models HTTP/1.1" 200 OK
2023-12-28 00:47:31 | INFO | controller | Register a new worker: http://localhost:40000
2023-12-28 00:47:31 | INFO | controller | Register done: http://localhost:40000, {'model_names': ['FERRET-13B-v0'], 'speed': 1, 'queue_length': 0}
2023-12-28 00:47:31 | INFO | stdout | INFO: 127.0.0.1:45318 - "POST /refresh_all_workers HTTP/1.1" 200 OK
2023-12-28 00:47:31 | INFO | stdout | INFO: 127.0.0.1:45324 - "POST /list_models HTTP/1.1" 200 OK
2023-12-28 00:47:35 | INFO | controller | Receive heart beat. http://localhost:40000
2023-12-28 00:47:35 | INFO | stdout | INFO: 127.0.0.1:41430 - "POST /receive_heart_beat HTTP/1.1" 200 OK
2023-12-28 00:47:43 | INFO | controller | names: ['http://localhost:40000'], queue_lens: [0.0], ret: http://localhost:40000
(ferret) root@58dfc909b9e7:/share/ml-ferret# python -m ferret.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --add_region_feature --port 8501
[2023-12-28 00:47:22,094] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2023-12-28 00:47:23 | INFO | gradio_web_server | args: Namespace(host='0.0.0.0', port=8501, controller_url='http://localhost:10000', concurrency_count=8, model_list_mode='reload', share=False, moderate=False, embed=False, add_region_feature=True)
2023-12-28 00:47:23 | INFO | gradio_web_server | Models: ['FERRET-13B-v0']
2023-12-28 00:47:23 | INFO | gradio_web_server | Namespace(host='0.0.0.0', port=8501, controller_url='http://localhost:10000', concurrency_count=8, model_list_mode='reload', share=False, moderate=False, embed=False, add_region_feature=True)
2023-12-28 00:47:23 | ERROR | stderr | /root/miniconda3/envs/ferret/lib/python3.10/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Textbox, please remove them: {'container': False}
2023-12-28 00:47:23 | ERROR | stderr | warnings.warn(
2023-12-28 00:47:25 | ERROR | stderr | /root/miniconda3/envs/ferret/lib/python3.10/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Dropdown, please remove them: {'container': False}
2023-12-28 00:47:25 | ERROR | stderr | warnings.warn(
2023-12-28 00:47:25 | INFO | stdout | Running on local URL: http://0.0.0.0:8501
2023-12-28 00:47:25 | INFO | stdout |
2023-12-28 00:47:25 | INFO | stdout | To create a public link, set `share=True` in `launch()`.
2023-12-28 00:47:31 | INFO | gradio_web_server | load_demo. ip: 172.21.0.1
2023-12-28 00:47:31 | INFO | gradio_web_server | Models: ['FERRET-13B-v0']
2023-12-28 00:47:34 | INFO | stdout | Init Uploading Images.
2023-12-28 00:47:43 | INFO | gradio_web_server | add_text. ip: 172.21.0.1. len: 13
2023-12-28 00:47:43 | INFO | stdout | No location, copy original image in add_text
2023-12-28 00:47:43 | INFO | gradio_web_server | http_bot. ip: 172.21.0.1
2023-12-28 00:47:43 | INFO | gradio_web_server | model_name: FERRET-13B-v0, worker_addr: http://localhost:40000
2023-12-28 00:47:43 | INFO | stdout | Input Image Size:(512, 512)
2023-12-28 00:47:43 | INFO | stdout | Input Image Size:(512, 512)
2023-12-28 00:47:43 | INFO | gradio_web_server | ==== request ====
{'model': 'FERRET-13B-v0', 'prompt': 'A chat between a human and an AI that understands visuals. In images, [x, y] denotes points: top-left [0, 0], bottom-right [width-1, height-1]. Increasing x moves right; y moves down. Bounding box: [x1, y1, x2, y2]. Image size: 1000x1000. Follow instructions. USER: <image>\nwhat is this? ASSISTANT:', 'temperature': 0.2, 'top_p': 0.7, 'max_new_tokens': 512, 'stop': '</s>', 'images': "List of 1 images: ['195205623896f712c8831c15be32a339']"}
2023-12-28 00:47:43 | INFO | gradio_web_server | ==== add region_masks_in_prompts to request ====
2023-12-28 00:47:43 | INFO | stdout | Input Image Size:(512, 512)
2023-12-28 00:47:43 | INFO | stdout | Input Prompt: A chat between a human and an AI that understands visuals. In images, [x, y] denotes points: top-left [0, 0], bottom-right [width-1, height-1]. Increasing x moves right; y moves down. Bounding box: [x1, y1, x2, y2]. Image size: 1000x1000. Follow instructions. USER: <image>
2023-12-28 00:47:43 | INFO | stdout | what is this? ASSISTANT:
(base) root@58dfc909b9e7:/share/ml-ferret# conda activate ferret
(ferret) root@58dfc909b9e7:/share/ml-ferret# CUDA_VISIBLE_DEVICES=0 python -m ferret.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./checkpoints/FERRET-13B-v0 --add_region_feature
[2023-12-27 16:29:26,569] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2023-12-27 16:29:27 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40000, worker_address='http://localhost:40000', controller_address='http://localhost:10000', model_path='./checkpoints/FERRET-13B-v0', model_base=None, model_name=None, multi_modal=False, keep_aspect_ratio=False, num_gpus=1, limit_model_concurrency=5, stream_interval=1, no_register=False, load_8bit=False, load_4bit=False, add_region_feature=True, image_w=336, image_h=336)
2023-12-27 16:29:27 | INFO | model_worker | Loading the model FERRET-13B-v0 on worker 9be48e ...
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███████████████████████████▎ | 1/3 [00:19<00:39, 19.54s/it]
Loading checkpoint shards: 67%|██████████████████████████████████████████████████████▋ | 2/3 [00:35<00:17, 17.59s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████| 3/3 [00:45<00:00, 13.85s/it]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████| 3/3 [00:45<00:00, 15.06s/it]
2023-12-27 16:30:14 | ERROR | stderr |
2023-12-27 16:30:16 | INFO | model_worker | Register to controller
2023-12-27 16:30:16 | ERROR | stderr | INFO: Started server process [243]
2023-12-27 16:30:16 | ERROR | stderr | INFO: Waiting for application startup.
2023-12-27 16:30:16 | ERROR | stderr | INFO: Application startup complete.
2023-12-27 16:30:16 | ERROR | stderr | INFO: Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit)
2023-12-27 16:31:04 | INFO | stdout | INFO: 127.0.0.1:46046 - "POST /worker_get_status HTTP/1.1" 200 OK
2023-12-27 16:31:18 | INFO | stdout | INFO: 127.0.0.1:33794 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-12-27 16:31:18 | INFO | model_worker | Add region_masks to image_args.
2023-12-27 16:37:48 | INFO | stdout | INFO: 127.0.0.1:38944 - "POST /worker_get_status HTTP/1.1" 200 OK
2023-12-27 16:37:59 | INFO | stdout | INFO: 127.0.0.1:56904 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-12-27 16:38:00 | INFO | model_worker | Add region_masks to image_args.
2023-12-28 00:46:20 | INFO | model_worker | Register to controller
2023-12-28 00:47:23 | INFO | stdout | INFO: 127.0.0.1:43410 - "POST /worker_get_status HTTP/1.1" 200 OK
2023-12-28 00:47:31 | INFO | stdout | INFO: 127.0.0.1:43426 - "POST /worker_get_status HTTP/1.1" 200 OK
2023-12-28 00:47:43 | INFO | stdout | INFO: 127.0.0.1:48710 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2023-12-28 00:47:43 | INFO | model_worker | Add region_masks to image_args.
@bensonbs Your logs look fine to me. It seems model_worker can successfully receive input since Add region_masks to image_args is printed and the generation process doesn't raise any error. I also tried your commands in my sever and it works well.
Can you try to (1). make sure the model is selected in the model list of demo webpage (as highlighted in the figure) (2). click the regenerate button. (3). refresh the demo webpage (4). try different ports in case there is a conflict with other processes.
@bensonbs Your logs look fine to me. It seems model_worker can successfully receive input since
Add region_masks to image_argsis printed and the generation process doesn't raise any error. I also tried your commands in my sever and it works well. Can you try to (1). make sure the model is selected in the model list of demo webpage (as highlighted in the figure) (2). click the regenerate button. (3). refresh the demo webpage (4). try different ports in case there is a conflict with other processes.
After a whole day of trying, I think it's 'doge' causing the trouble.🫠
root@46290910f996:~# python3 -m ferret.model.apply_delta --base ./model/vicuna-13b-v1-3 --target model/ferret-13b-v1-3 --delta model/ferret-13b-delta
[2023-12-28 10:44:30,138] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loading base model
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/ferret/model/apply_delta.py", line 69, in <module>
apply_delta(args.base_model_path, args.target_model_path, args.delta_path)
File "/root/ferret/model/apply_delta.py", line 36, in apply_delta
base = AutoModelForCausalLM.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 441, in from_pretrained
config, kwargs = AutoConfig.from_pretrained(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 926, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 632, in __getitem__
raise KeyError(key)
KeyError: 'llava'
root@46290910f996:~# python3 -m ferret.model.apply_delta --base ./model/vicuna-13b-v1-3 --target model/ferret-13b-v1-3 --delta model/ferret-13b-delta [2023-12-28 10:44:30,138] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect) Loading base model Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/root/ferret/model/apply_delta.py", line 69, in <module> apply_delta(args.base_model_path, args.target_model_path, args.delta_path) File "/root/ferret/model/apply_delta.py", line 36, in apply_delta base = AutoModelForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 441, in from_pretrained config, kwargs = AutoConfig.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 926, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 632, in __getitem__ raise KeyError(key) KeyError: 'llava'
I also got this error when I tried to use a pre-trained llava model, and I tried changing the model type in config.json from "llava" to "llama", and its works for me
root@46290910f996:~# python3 -m ferret.model.apply_delta --base ./model/vicuna-13b-v1-3 --target model/ferret-13b-v1-3 --delta model/ferret-13b-delta [2023-12-28 10:44:30,138] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect) Loading base model Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/root/ferret/model/apply_delta.py", line 69, in <module> apply_delta(args.base_model_path, args.target_model_path, args.delta_path) File "/root/ferret/model/apply_delta.py", line 36, in apply_delta base = AutoModelForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 441, in from_pretrained config, kwargs = AutoConfig.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 926, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 632, in __getitem__ raise KeyError(key) KeyError: 'llava'
@bensonbs It seems that your base model (./model/vicuna-13b-v1-3) actually has its "model_type" in config.json as 'llava'. Can you double-check your vicuna weights? It should use "llama" as its "model_type". When applying delta, we don't need any weights from LLaVA and only need the original vicuna.
i got error while i was running the codes for checkpoints:
python3 -m ferret.model.apply_delta
--base /model/vicuna-7b-v1-3
--target /model/ferret-7b-v1-3
--delta /model/ferret-7b-delta
the error is: pickle.UnpicklingError: invalid load key, 'v'.
i've placed vicuna-7b-v1-3 and ferret-7b-delta both under model folder. i couldnt find ferret-7b-v1-3 so i created an empty folder with the same name. how can i get pass this error?
i got error while i was running the codes for checkpoints:
python3 -m ferret.model.apply_delta --base /model/vicuna-7b-v1-3 --target /model/ferret-7b-v1-3 --delta /model/ferret-7b-delta
the error is: pickle.UnpicklingError: invalid load key, 'v'.
i've placed vicuna-7b-v1-3 and ferret-7b-delta both under model folder. i couldnt find ferret-7b-v1-3 so i created an empty folder with the same name. how can i get pass this error?
@crabmon Hi, placing vicuna and ferret-delta in the same model folder and creating an empty folder of ferret-7b-v1-3 works well from my side. Can you provide more details of the error? such as screenshots of command/error.
hi @Hxyou, thank you for your reply. i managed to get pass that step. now im stucked again. it says NETWORK error due to HIGH traffic.....
from terminal, it says caught unknown error CUDNN error: cudnn_status_internal_error.
any clue?
hi @Hxyou, thank you for your reply. i managed to get pass that step. now im stucked again. it says NETWORK error due to HIGH traffic.....
from terminal, it says caught unknown error CUDNN error: cudnn_status_internal_error.
any clue?
@crabmon Can you show us a screenshot of the terminal errors? and What is the GPU you are using? What is the version of pytorch and cuda?
hi @Hxyou, thank you for your reply. i managed to get pass that step. now im stucked again. it says NETWORK error due to HIGH traffic..... from terminal, it says caught unknown error CUDNN error: cudnn_status_internal_error. any clue?
@crabmon Can you show us a screenshot of the terminal errors? and What is the GPU you are using? What is the version of pytorch and cuda?
@Hxyou , please refer to the screen capture before.
im using 3090 , Cuda version 12 and Driver Version 525.147.05
@crabmon CUDNN_STATUS_INTERNAL_ERROR is typically hard to debug since it gives no real error message. But in many cases, it's due to OOM. 3090 only has 24GB memory and a 13B model often consumes more than 20GB memory. Are you trying a 7b or 13b model?
Hi there @Hxyou ,
I was wondering about the dataset.
Right I can only find a few datapoints in the eval set, which I assume at least have the same format as the rest of the dataset (?).
I can see that some loading of different datasources is happening in https://github.com/apple/ml-ferret/blob/main/ferret/train/train.py, but since the repos main README explicitly mentions that the dataset has it's own license, I was expecting that I can download it as is somewhere.
Is there a timeline on when it will be released publicly? Or is there a link to download it somewhere or a script to basically reconstruct it using a standalone script and download instructions for the other datasets it is based on?
Thanks!
