John Sangwon Suh

As an addendum, as they state it, the compiler is not threadsafe, so you will need to use a lock when initializing the Compiler and compiling lock = threading.Lock() def...

@XHPlus There's a lot of open source models on HuggingFace driven by https://huggingface.co/TheBloke. Many people in the open source community use those quantized models on TGI / vLLM.