implementation of a leaderboard for different quantizations (gguf 4 vs 6 vs etc bits)

[chakravarthik27]

Summary:

This issue proposes the implementation of a leaderboard to compare the performance of different quantization settings (e.g., GGUF 4 bits, GGUF 6 bits, etc.) within LangTest. This leaderboard would allow users to easily identify the most effective quantization settings for their specific needs and usage scenarios.

Motivation:

• Quantization is a technique used to reduce the size of a model by lowering the precision of its weights and activations. This can be beneficial for reducing storage requirements and improving inference speed on resource-constrained devices.
• LangTest supports various quantization settings, but it currently lacks a mechanism to directly compare the performance of these settings.
• A leaderboard would provide valuable insights into the trade-offs between model size, inference speed, and accuracy for different quantization configurations.

Proposed solution:

• Implement a leaderboard that displays the performance of different quantization settings on a set of LangTest benchmarks.
• The leaderboard should include the following information for each quantization setting:
  • Quantization configuration (e.g., GGUF 4 bits, GGUF 6 bits)
  • Model size
  • Inference speed
  • Accuracy on LangTest benchmarks
• The leaderboard should allow users to filter and sort results based on different criteria (e.g., model size, inference speed, accuracy).

Additional considerations:

• The specific benchmarks used in the leaderboard should be clearly defined and relevant to the target use cases of LangTest.
• The leaderboard should be visually appealing and easy to interpret for users.
• The implementation should be modular and extensible to accommodate future additions of new quantization settings or benchmarks.

Benefits:

• The proposed leaderboard would empower users to make informed decisions about quantization settings for their LangTest applications.
• It would facilitate the sharing and comparison of best practices for LangTest quantization.
• It would promote further research and development in the area of quantization for language models.