transformerlab
custom app integration possible?
is there any way to access the fine-tuning features programmatically? I want to prepare the dataset and template and just call start_finetuning() or something
Transformer Lab is made up of a React app front end (transformerlab-app) talking to a python engine backend (transformerlab-api) that does all of the work. So most things that the app does are possible to do programmatically via API.
If you are running Transformer Lab on your computer you can see the API docs by going to http://localhost:8338/docs .
e.g. In the case of fine-tuning, you need to first create a template for a fine-tune (doing this in the app might be easier). Then if you want to trigger a fine-tune you just need to create a new job with that template (you can do this over and over, possibly updating the template between calls).
If you're interested in doing this i would recommend joining out discord and somebody can maybe help you with any unclear details (the docs are quite lean).
We are also working on new workflow features that will make this more flexible...coming soon!
Hi @itsPreto, Its possible what we might have multiple ways to go about this including the ones Tony mentioned. Could you provide a more detailed explanation of the use case and I'll help you out!
Hey guys, first off I just want to extend my kudos because the project is sweet! I've used it several times for experiments on my m1 and love how accessible it is.
I’m working on an Ollama-based system that: • Ingests archived chat data (from GPT, Claude, etc.) • Transforms it into fine-tune-ready datasets based on user-defined goals
With goals to: • Establish an easy workflow that takes users from their raw chat data to a fine tuned model-- ideally being guided by an LLM/Agent through the process.
Some example use cases would be: • Personal therapist model that learns how to guide users through issues without enabling bad habits • Dev assistant model fine-tuned my personal coding practices and patterns.
Essentially-- 1. Raw chat logs → structured dataset 2. Apply template → trigger fine-tune via API 3. Return model/artifacts
Is there a recommended way to automate this using your backend? Also, curious how upcoming workflow features might support this.
Thanks for the great work.