quickstart basic - missing qa/llm_tests:?

[richardehughes]

Ran: llmtune generate config llmtune run ./config.yml

Things worked well (once I fixed my mistake with Mistral/huggingface repo permissions). The job ran very fast and put results into the "experiment" directory. But the experiment/XXX/results/ directory only has a "results.csv" file in it. I expected there to be results from the qa/llm_tests section in the config.yml file, which looks like this: qa: llm_tests: - jaccard_similarity - dot_product - rouge_score - word_overlap - verb_percent - adjective_percent - noun_percent - summary_length

Do I have to do something extra to get the qa to run?

[benjaminye]

Hi @richardehughes, we decided to take little pause in adding new features to the CLI until we finish our test suite (for the program, that is). This should be done by today -- and we'll release the LLM tests integration under a new minor version next week.

But you can still run all the test by importing the module directly like so:

from src.qa.qa import LLMTestSuite
from src.qa.qa_tests import LengthTest, JaccardSimilarityTest

# Load prompts, ground_truths, and model_preds from the results.csv file
# Your code here

# Running LLM Tests
tests = [LengthTest(), JaccardSimilarityTest()]
test_suite = LLMTestSuite(tests, prompts, ground_truths, model_preds)
test_suite.run_tests()
test_suite.print_test_results()
test_suite.save_test_results("path/to/save/test_results.csv")

[benjaminye]

This is addressed in release v0.2.2

https://github.com/georgian-io/LLM-Finetuning-Toolkit/releases/tag/v0.2.2