llamafile icon indicating copy to clipboard operation
llamafile copied to clipboard

API endpoint to return llamafile version + other metadata

Open k8si opened this issue 11 months ago • 1 comments

It would be useful to have an API endpoint like /info that would return a json dict containing 1) the version of llamafile and 2) model metadata/configuration.

Llamafile version allows me to adapt client code to work with each release. E.g. I think the output of the /embedding endpoint changed between release v0.6.2 and what's currently on master. This will be a breaking change for my LlamaIndex integration -- v0.6.2 llamafiles will work but v0.6.3 llamafiles will not (in certain cases).

It would also be useful to have a way to get model metadata via an endpoint. E.g. user systems need to know the model's max sequence length in order to truncate/batch prompts appropriately. Currently there's no way to get this info.

k8si avatar Mar 07 '24 18:03 k8si