werruww

[bamba](https://github.com/werruww/HIGGS/blob/main/bamba_9bgguf%20(1).ipynb)

how to run Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 on colab t4 ram12 viga 16

```python from huggingface_hub import snapshot_download import torch from accelerate import infer_auto_device_map from transformers import AutoModelForCausalLM, AutoConfig checkpoint = "marcsun13/gpt2-xl-linear-sharded" weights_location = snapshot_download(repo_id=checkpoint) # Instead of loading directly from checkpoint, use...

If I have a single 16 GB Vega and a processor, how do I run a larger model of Vega on the Vega and the processor so that I can...

What are the steps from a to z to run a model larger than the 16 GB Vega on the Vega and the processor? Starting from downloading the model, then...

from huggingface_hub import snapshot_download checkpoint = "marcsun13/gpt2-xl-linear-sharded" weights_location = snapshot_download(repo_id=checkpoint) from accelerate import init_empty_weights from mingpt.model import GPT model_config = GPT.get_default_config() model_config.model_type = 'gpt2-xl' model_config.vocab_size = 50257 model_config.block_size = 1024...

clear device_map = infer_auto_device_map( model, max_memory={0: "10GiB", "cpu": "10GiB"} ) and pot device_map="auto"

If you allow me to write a complete code that I trust I copied the codes from the site randomly If it is possible to write a complete code that...

If possible, a collab page tpu 24g A model larger than 24 GB To clarify things Thank you

ValueError Traceback (most recent call last) [](https://localhost:8080/#) in () 1 from accelerate import load_checkpoint_and_dispatch 2 ----> 3 model = load_checkpoint_and_dispatch( 4 model, checkpoint=weights_location, device_map="auto", no_split_module_classes=['Block'] 5 ) 2 frames [/usr/local/lib/python3.10/dist-packages/accelerate/utils/modeling.py](https://localhost:8080/#)...