question: how would you go about saving a control vector for later use

[vpicone]

Perhaps a naive question, but rather than training a control vector with each run. How might I go about saving it for inference later?

[vgel]

You can export it for use with llama.cpp with export_gguf, but I haven't written a method to read that gguf file back in for HuggingFace use yet.

In the meantime, you should be able to use np.save to save the dataclass as a dictionary, and then reconstitute it that way:

import dataclasses
import numpy as np
...
v = ControlVector.train(...)
np.save("vector.npy", dataclasses.asdict(v))

# later...
v = ControlVector(**np.load("vector.npy", allow_pickle=True).tolist())

Hope this helps!

[l1mc]

Hi @vgel,

Thanks for the helpful insight on saving a control vector for later use. I noticed that you mentioned the ability to export a control vector with export_gguf for use with llama.cpp. Could you provide an example or guide on how to perform this export?

I found a tutorial on How to convert HuggingFace model to GGUF format. However, I found it difficult to apply this method described here in a Jupyter Notebook (for example, the tutorial ipynb you provided).

Thanks!

[vgel]

@l1mc

Hi @vgel,

Thanks for the helpful insight on saving a control vector for later use. I noticed that you mentioned the ability to export a control vector with export_gguf for use with llama.cpp. Could you provide an example or guide on how to perform this export?

I found a tutorial on How to convert HuggingFace model to GGUF format. However, I found it difficult to apply this method described here in a Jupyter Notebook (for example, the tutorial ipynb you provided).

Thanks!

Sorry, didn't see this earlier! Once you have a vector trained with (e.g.) vector = ControlVector.train(...), you can simply export it with:

vector.export_gguf("vector.gguf")

If you're running a version with #34 applied, you can also then import the vector back to Python with

vector = ControlVector.import_gguf("vector.gguf")

or you can use it with llama.cpp using the ./main runner:

$ ./main ... --control-vector vector.gguf --control-vector-layer-range 14 26 ...

Hope this helps!

[vgel]

Closing this as addressed by #34