giskard
giskard copied to clipboard
Testset generation lost when a single generation fail
Issue Type
Bug
Source
source
Giskard Library Version
2.15.0
OS Platform and Distribution
Macos
Python version
3.12.4
Installed python packages
No response
Current Behaviour?
When generating a testset, it takes around 45-60 minutes to go thru a doc and produce the topics.
Then it starts generating the testset but... it's failing around 80% because of JSONDecode error and I have to restart again, including all the topic generation part.
We should implement a solution to save the progress, since it is very likely to have error like this when generating testset from very long docs.
Standalone code OR list down the steps to reproduce the issue
TODO
Relevant log output
No response