Using DSPy with no internet on Kaggle notebook?

[augustfr]

Trying to run a model with DSPy on a kaggle notebook with internet access turned off. Any suggestions?

Ideally I want to use Mixtral 8x7b. I have this implemented:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "/kaggle/input/mixtral/pytorch/8x7b-instruct-v0.1-hf/1"
tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id)

Unsure as to how to use this model with DSPy.

[arnavsinghvi11]

Hi @augustfr , you can try using HFModel to locally host models but this is not fully supported like the other clients.

[augustfr]

@arnavsinghvi11 that's what I've been trying to do now it's just then I have to first download the model from huggingface and move it into the huggingface cache on the Kaggle notebook first which is a bit hacky. Since if you just use HFModel it first tries to download the model from HF which doesn't work if the Kaggle notebook has no internet connection.

Let me know if you think this makes sense (the cache strategy). I just wonder if there's a way to use the Kaggle models directly instead of having to download from huggingface and import the .zip file into kaggle as a dataset.