patrickjchen

Is this the correct model after downloading from Huggingface? >>> model GemmaForCausalLM( (model): GemmaModel( (embed_tokens): Embedding(256000, 3072) (layers): ModuleList( (0-27): 28 x GemmaDecoderLayer( (self_attn): GemmaSdpaAttention( (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False) (k_proj):...

Dan, seems the implementation of save_pretrained/from_pretrained have some issues for the Gemma 7b model. My code wroks for Mistral. However, I felt Gemma version is a lot better.

and from the code: python3.11/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 193: if (param_name + ".quant_state.bitsandbytes__fp4" not in state_dict) and ( param_name + ".quant_state.bitsandbytes__nf4" not in state_dict ): raise ValueError( f"Supplied state dict for {param_name}...

Seems some keys are lost after reading back (from_pretrained()). There were 1234 keys, but after store/retrieve, there were only 1050 keys. For the top level state_dict

@danielhanchen Hi Dan, after I did model, tokenizer = FastLanguageModel.from_pretrained( model_name = "****", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) How to save the model to a...

Hi Dan, anyway to get around this error before it is fixed?

Thanks! It works!

Running again, got this error: ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject When running: model, tokenizer = FastLanguageModel.from_pretrained( model_name =...

Thanks, Dan. Yeah, I need to first download the packages , then install them without internet connection. Is that torch 2.2.2 is used by the notebook? It's implicitly installed when...