stevenlu

I'm not sure if I understood your question correctly. Firstly, AIlice has built-in support for some open-source models and the OpenAI API, but it cannot share the backend language model...

In AIlice, the model used is specified through command-line parameters (--modelID), where the ID has a prefix of "oai:" or "hf:", representing whether it is from OpenAI or Hugging Face....

My apologies, currently setting the modelID to file:... is not an official feature. You need to manually add this modelID in ALLMMeta.py for it to work. Please disregard my previous...

This is likely due to Windows not supporting zmq's ipc communication. You can try changing all instances of ipc:///... to tcp://127.0.0.1:5910x(choose a port as your prefer) in the config.json file....

Hey, go get the latest code of the master branch. I have run it in the local windows environment successfully. It is best to delete the previous config.json first. Regarding...

In addition, in order to allow scripts to execute code, WSL still needs to be installed on Windows (I think you have already installed it). We do not yet have...

The "Resource temporarily unavailable" error could be caused by the "python3" you used in "cmd", I replaced it with "python" and it worked. Running AIlice on Windows without WSL would...

This is a weird situation. It seems like the underlying package, transformers, can not recognize the mixtral model. If you are still interested on this issue, you can try another...

ailice_main --modelID=hf:Open-Orca/Mistral-7B-OpenOrca --prompt="main" --quantization=8bit --contextWindowRatio=0.6 There is no open source model that can really do this job, and this model is relatively the best.

Theoretically you can put it all in memory, but it will be very slow. In the future I will consider integrating faster inference engine, but the current 8GB GPU memory...