Consider Using Standard Config Format

[nazimamin]

Thank you for your work, this is great and will be very helpful for the OSS community. The custom configuration file named "Modelfile" works well in the context of this project.

I would like to discuss the possibility of using a standardized config format such as JSON5, TOML, YAML, or another similar standard. Those are battle-tested, easy to read and maintain, and have extensive tooling support. Using custom config language makes it harder for people to adapt quickly and makes it hard to build tooling support on top of it. Thoughts?

[mchiang0610]

Hey @nazimamin, thank you so much for submitting this, and bringing it to our attention.

Having worked with multiple config formats in the past, I think you and I know very well, no matter what we choose, there will be different trade offs. Some cater towards composability, and making it easy to generate via generator tools. Some are more human-readable than others, and the list can go on.

We probably won't get it right all the time, but we'll always pick the one that can address the biggest number of people. Always listening to more use-cases, and suggestions to determine what's the best.

We definitely should have good support to tools/formats that to enable use of Ollama's API.

[nkoehring]

I support the OPs suggestion. Maybe "getting it right" issues could be mitigated by supporting multiple formats that can be easily serialized. I for example find TOML very easy to read and write, but also simple to parse. Others might prefer YAML (which I don't like that much). Parsing those formats into the same memory structure should be rather easy, nowadays.

[pdevine]

I think there's potentially a misunderstanding of the purpose of the Modelfile. It's not a config file, but more akin to a Makefile or a Dockerfile. It's probably hard to see at this point because everything is still so rudimentary, but the vision is to have a format which can handle assembling all of the quantization levels, embeddings, and even peft/lora training all in the same file. Additionally, with the support of manifest lists in the model image we can support multiple weights formats beyond ggml v3. The point is that you can use the Modelfile to assemble your model, and use ollama push to push it to the registry where you can easily share it with other people.

So it's not so much as a config file as it is a script or a source file for assembling models. I appreciate the suggestion here, guys, but I'm going to go ahead and close the issue. There will be ollama config files at some point for tweaking parameters in the ollama runner/server, and those will definitely be normal config files.

[pdevine]

One thing I should also mention; you can use JSON to set the hyper-parameters already when calling POST /api/generate. We could fairly easily expose this through ollama run so you could easily tweak the parameters. LMK if this would be useful.

[nathanleclaire]

Yea, there is never any good answer on this kind of thing. Dockerfile took off like crazy because it's so easy to understand and read/write. And Kubernetes had loads of YAML with infinite flexibility, but it made us all miserable, and generation layers on top like Helm made everything worse. It might be a good compromise if Ollama had a well-defined API (or JSON compatibility with Modelfile, that type of thing), that didn't push so much of the magic to the client like Docker does in a lot of instances.

[sammcj]

I'd really like to see Ollama support using a standard config file for model server settings and default parameters.

Somewhat related #4120 #4154 #4123 #836

[chigkim]

This seems an old issue, so maybe neglected, but regardless it would be amazing to be able to set with config file like yaml/json format.

OLLAMA_HOST: "0.0.0.0"
OLLAMA_NOHISTORY: true
OLLAMA_FLASH_ATTENTION: true
OLLAMA_NUM_PARALLEL: 4
OLLAMA_MAX_LOADED: 2