giskard icon indicating copy to clipboard operation
giskard copied to clipboard

Testset generation lost when a single generation fail

Open kevinmessiaen opened this issue 5 months ago • 2 comments

Issue Type

Bug

Source

source

Giskard Library Version

2.15.0

OS Platform and Distribution

Macos

Python version

3.12.4

Installed python packages

No response

Current Behaviour?

When generating a testset, it takes around 45-60 minutes to go thru a doc and produce the topics. 

Then it starts generating the testset but... it's failing around 80% because of JSONDecode error and I have to restart again, including all the topic generation part.

We should implement a solution to save the progress, since it is very likely to have error like this when generating testset from very long docs.

Standalone code OR list down the steps to reproduce the issue

TODO

Relevant log output

No response

kevinmessiaen avatar Sep 04 '24 01:09 kevinmessiaen