Jake Ju

@doublexxking Nope, I failed to do so. But I rewrite the whole network manually by caffe2 and load weights from a trained PyTorch model. It works perfectly for me. I...

It seems that the GPU is not activated during restoring from checkpoint, right?

我后面用的是pytorch的libtorch C++库完成的部署，没有使用paddle

也正常吧，pointnet++在encoder部分叠加了多个pointnet，计算量天然比pointnet大，infer时间长也正常，你不放心可以排查下pointnet++中耗时比较高的瓶颈在哪儿，可以如何优化

Hi @m-bain, any plan to support `whisper-large-v3-turbo`?

> > Sorry for late reply. > > Currently, you can use `pipe.get_logits` to obtain the logits with seq_len x vocab_size shape. It's not thread safe, I'm not sure if...

I've encountered the same issue. Full-parameter fine-tuning works perfectly, but LoRA fine-tuning produces garbled results. I'm attempting to migrate my project from the zjysteven/lmms-finetune repository, where LoRA fine-tuning works correctly.

It seems that you have loaded the wrong vision tower ckpt. Can you provide the config.json and your input parameter for `merge_lora_weights.py`

@haozhang1234 You may take a look of the `build_vision_tower` function , [line#23](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/333d6fc705f8b62325c61fda70a629cdfcf54129/llava/model/multimodal_encoder/builder.py#L21) should be executed based on your config: `return SigLipVisionTower(vision_tower, vision_tower_cfg=vision_tower_cfg, **kwargs)`. Clearly, there is something wrong with the...

> In addition, have you checked the trainable params when do lora fine tune? I found that if your print out all trainable params at this line [llava/train/train.py#L1695](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/333d6fc705f8b62325c61fda70a629cdfcf54129/llava/train/train.py#L1695), just before...