Feilong Chen

icon indicating a website link https://phellonchen.github.io/

icon location Beijing, China icon location Institute of Automation, Chinese Academy of Sciences & University of Chinese Academy of Sciences

Results 7 comments of Feilong Chen
trafficstars

LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.

same issue

such as

I guess you could check the pytorch version. The code is tested on v0.3.0.

Pre-Training BLIP2 Log

I have modified the code for pretraining from scratch and do not load a pre-trained blip2 checkpoint. But the loss is still difficult to reduce. I just changed the part...

Pre-Training BLIP2 Log

``` \\ blip2_qformer load_finetuned = cfg.get("load_finetuned", True) load_pretrained = cfg.get("load_pretrained", True) if load_finetuned or load_pretrained: model.load_checkpoint_from_config(cfg) return model ``` stage 1 logs for 3m images {"train_lr": "0.000", "train_loss": "7.367"} {"train_lr":...

Pre-Training BLIP2 Log

Thanks. Could you tell me how much loss is a good convergence of stage 1 and stage 2? When I train the model for 10 epochs. the loss seems still...

Pre-Training BLIP2 Log

OK. I will try again and give back the results. Thank you for your patience.

Pre-Training BLIP2 Log

Yes. Thanks for the authors' excellent works. In the process of trying to train from scratch, I found that the language model I used at the beginning was too small...