Official Support for GGUF Quantization in BigCode Starcoder2 to Enhance Accessibility and Efficiency

[babycommando]

Dear BigCode team, what a wonderful project!

I am writing this feature request for official implementation of GGUF quantization for Starcoder2 to enhance its adoption with coding platforms and APIs such as Ollama and LMStudio.

Despite the model's advanced capabilities with its versions, its integration and usability in the OpenAI-API style coding ecosystem, including extensions like "Continue" for VSCode, could be significantly improved. The current lack of support for GGUF quantization limits its potential reach and utility.

An official implementation by your team would ensure optimal performance and compatibility, eliminating the need for community-driven workarounds. I urge you to consider this proposal as a step towards making BigCode Starcoder2 a more versatile and inclusive tool for the developer community. Official GGUF quantization could significantly impact its adoption and effectiveness across diverse development environments.

Thank you for your time and consideration of this important enhancement. I look forward to your positive response and the future success of BigCode Starcoder2.