Support for fine tuning GGML libraries

[andrewschreiber]

Describe the type of feature and its functionality.

GGML is an exciting new library that's being used for edge computing of ML models. For example, llama.cpp and whisper.cpp.

GGML does support fine-tuning, but usually it's by importing from another model format. For example, fine-tuning a HuggingFace model, then converting it as shown here: https://github.com/ggerganov/whisper.cpp/tree/master/models#fine-tuned-models

Seeing as Flower supports HuggingFace, perhaps there is an approach here? Curious the community's thoughts.

Describe step by step what files and adjustments are you planning to include.

An example pipeline of fine tuning a ggml model using flower.

Is there something else you want to add?

No response

[danieljanes]

Hi @andrewschreiber,

Thanks for reaching out. I'm quite excited about this - I actually commented on the GGML HN thread asking if it supported training on mobile platforms. The reply was that while training is possible, it's not documented yet.

Have you used GGML fine-tuning? Happy to collaborate on a code example with Flower if you're interested!

[andrewschreiber]

Thanks for the quick response! It's encouraging to see your enthusiasm as well @danieljanes

I expect to have bandwidth in early August to dive in. Privacy-respecting training is paramount in my new project and flower + ggml seems like the strongest angle of attack

[WilliamLindskog]

Hi @andrewschreiber

Thanks for raising this feature request. Is this something that you would like to implement, maybe open a PR?

[andrewschreiber]

Hey still quite busy with other efforts unfortunately. Will loop back when I have more availability.

[WilliamLindskog]

Sounds good! Let me know if there's anything we can help you with in the meantime.

[WilliamLindskog]

Open comment if anyone has seen similar implementations of GGML that can be replicated in Flower?

[WilliamLindskog]

We are looking into this, one comment would be, would it suffice to change C++ example to something like in this script?

#include "arg.h"
#include "common.h"
#include "log.h"
#include "llama.h"

The C++ example is not up-to-date but is work in progress, see here:

• https://github.com/adap/flower/issues/5286
• https://github.com/adap/flower/pull/5287

[milonimittal]

I find this very interesting, would like to contribute. Is there a possibility? Also, just to make sure my understanding is right: We are looking to create a pipeline where a model is FL trained using Flower and then the model is converted to a ggml format so that the inference can happen at edge. Is that understanding right?

I am happy to pick this or collaborate on this.