Query Regarding Minimum Hardware Requirements for Fine-tuning and Inference

[martinb-ai]

Hi there,

Firstly, I want to express my appreciation for the insightful tutorial and the fine-tuning repository. I've found them extremely useful. :rocket:

I'm looking to clarify what the minimum computer hardware requirements are for fine-tuning and inference with the models supported in this repo. I encountered some out-of-memory (OOM) issues during quantization on a system with 8GB RAM running on a CPU only.

The reason I'm asking this is because I'm considering using this repo for our open-source project (OpenBBTerminal). Understanding the minimum requirements will help us ensure the widest possible user accessibility.

Thanks in advance for your help on this matter.

[ArturK-85]

Hello, I trained the Adapter correctly using the RTX3060 with 12GB VRam and the StableLM-3B model. Inference uses about 7.5GB and in my opinion the card I use is the minimum for the 3B model. 8GB may not be enough. Quantized 7B llm.8bit model may need ~9.5GB of memory and ~5GB for gptq.4bit. I hope I helped, please ask if the answer is not sufficient

[martinb-ai]

Hi @ArturK-85 Thank you for your response. That is good to hear.

I just swapped to a different machine with identical specs as you: RTX3060 with 12GB VRAM.

Running the following worked only when creating a swap file for RAM purposes.

python scripts/download.py --repo_id togethercomputer/RedPajama-INCITE-Base-3B-v1
python scripts/convert_hf_checkpoint.py --checkpoint_dir checkpoints/togethercomputer/RedPajama-INCITE-Base-3B-v1

[ArturK-85]

@martinb-bb Finetune_adapter is possible on this equipment, full training is probably not possible. For me, finetuning did not use swap, except for the card I have 64GB DDR Ram. What value of micro_batch_size have you set? It should be 4 for this card. And i use Alpaca dataset

[martinb-ai]

@ArturK-85 Ah so 64GB is needed. I only have 16GB of DDR on this machine. Perhaps the issue is there.

[ArturK-85]

@martinb-bb 16GB is enough, i observed max system ram usage ~4.5GB, when finetune adapter ram usage increase to ~10GB. What parameter for training you use? What dataset? Memory usage also depends on the size of the dataset, micro_batch_size etc...

[carmocca]

Quantization support landed with #104 so now you can do inference with less requirements.